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1  Introduction  and  Project  Overview 

Twenty-five  years  ago,  a  small  city  on  the  California  coast  had  a  serious 
problem.  Nestled  between  the  cold  Pacific  Ocean  and  the  San  Gabriel 
Mountains,  producing  one  of  the  most  pleasant  climates  in  the  world,  the 
City  of  Santa  Barbara  had  become  a  magnet  for  mobility-impaired  and 
visually-impaired  individuals,  who  were  attracted  to  the  area  by  a  lack  of 
inclement  weather,  a  functional  and  well-supported  public  transit  system, 
a  vibrant  technology-  and  tourism-based  economy,  and  significant  public- 
assistance  housing  and  support  programs. 

A  growing  cohort  of  students  with  mobility  and  vision  impairments  like¬ 
wise  were  attracted  to  the  University  of  California,  Santa  Barbara  (UCSB) 
because  of  the  flat  terrain,  lack  of  inclement  weather,  and  general  support 
for  accessibility.  Unfortunately,  during  this  period  the  university  was  also 
the  center  of  considerable  controversy  as  thousands  of  students,  faculty, 
and  staff  drove,  hiked,  and  walked  to  work  or  class  every  day.  Injuries,  ac¬ 
cidents,  and  frustration  were  commonplace,  as  bike  paths  and  walkways 
crossed  busy  roads,  and  infrastructure  was  being  expanded  to  accommo¬ 
date  the  demands  of  students,  faculty,  and  staff,  who  felt  that  the  Universi¬ 
ty  should  have  the  best  facilities  to  support  their  growing  ambition  and 
reputation.  The  chaotic  mass  of  pedestrians,  bicycles,  skateboards,  and 
motorized  vehicles  were  mixed  together  in  a  somewhat  inadequate  side¬ 
walk  and  road  network. 

Campus  planning  authorities  realized  that  they  would  need  to  bring  order 
to  the  chaos  and  they  embarked  on  a  long-term,  multi-faceted  effort  to 
partition  the  incompatible  modes  of  transportation  into  separate  transpor¬ 
tation  networks,  and  to  provide  a  buffer  for  the  pedestrians,  who  counted 
among  themselves  hundreds  of  students,  faculty,  and  staff  with  vision  and 
mobility  impairments. 
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Figure  1.  Reginald  Golledge 
using  the  UCSB  Personal 
Guidance  System,  circa 
2003 


An  unfortunate  yet  concurrent  event 
during  this  time  period  was  the  com¬ 
plete  loss  of  vision  by  a  well-known 
UCSB  Geographer  and  faculty  member, 
Reginald  Golledge,  who  had  been  an 
able-bodied  and  rugged  young  adult. 

His  sudden  and  unexplained  loss  of  vi¬ 
sion  was  a  devastating  blow  to  what  had 
been  a  very  promising  career  as  a  geog¬ 
raphy  faculty  member,  where  vision, 
and  the  ability  to  visually  interact  with 
maps  and  computers  was  at  the  very 
foundation  of  his  identity  and  the  iden¬ 
tity  of  the  local  department.  Fortunately 
for  Dr.  Golledge,  two  faculty  members 
from  the  UCSB  Department  of  Psychol¬ 
ogy,  Dr.  Jack  Loomis  and  Dr.  Roberta 
Klatzky  came  up  with  a  conceptual  de¬ 


sign  for  a  system  that  would  allow  Dr.  Reginald  Golledge  to  use  a  mobile 
geographic  information  system  complete  with  GPS,  auditory  cues,  and  di¬ 
rectional  cues,  to  regain  mobility  and  navigate  across  the  increasingly  busy 
campus,  and  therefore  regain  his  ability  to  move  and  interact  within  his 
workplace. 


The  development,  testing,  and  refinement  of  this  Personal  Guidance  Sys¬ 
tem  (Figure  l),  described  first  by  Loomis  in  19851  and  later  in  Golledge  et 
al.  (1998)  and  Loomis  et  al.  (2001),  became  a  joint  research  effort  by  no 
fewer  than  30  researchers,  staff,  and  post-doctoral  researchers  and  ex¬ 
tended  from  its  beginnings  in  the  mid-1980s  to  the  year  2009,  coinciding 
with  Dr.  Reginald  Golledge’s  death.  The  significant  body  of  research  asso¬ 
ciated  with  the  system  design,  testing,  refinement,  and  usability,  includes 
more  than  40  peer-reviewed  publications,  and  hundreds  of  conference 
presentations  and  public  demonstrations.2 


1  See  "Re:  Digital  Map  and  Navigation  System  for  the  Visually  Impaired,”  UCSB,  November  1, 
2012http://www.geog.ucsb.edu/pgs/papers/loomis  1985.pdf 

2  The  UCSB  Personal  Guidance  System  Project  research  team  produced  42  peer-reviewed  technical  and 
basic  research  publications  between  1985  and  2008  about  system  testing,  design,  and  usability.  See 
“Publications,"  UCSB,  November  1,  2012,  http://www.geog.ucsb.edu/pgs/publications.htm 
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The  UCSB  Personal  Guidance  System  demonstrated  the  capability  of  ge¬ 
otechnology  to  make  a  significant  personal  impact  in  the  lives  of  the  visu¬ 
ally-  and  mobility-impaired,  and  addresses  some  very  significant  issues 
related  to  spatial  behavior,  wayfinding,  spatial  cognition,  and  human- 
computer  interaction.  The  system’s  most  notable  shortcoming  was  its  ina¬ 
bility  to  accommodate  real-time  updates  and  transitory  events  and  obsta¬ 
cles  that  significantly  impact  navigation  and  wayfinding  for  blind,  visually- 
and  mobility-impaired  individuals. 

Transitory  events  and  obstacles  include  any  object,  item,  or  condition  that 
impacts  navigation  and  wayfinding,  causing  the  blind,  visually-  or  mobili¬ 
ty-impaired  individuals  to  alter  their  planned  route.  In  some  cases,  these 
transitory  events  are  inconveniences,  requiring  slight  modification  to  a 
route,  while  in  other  situations  they  require  significant  back-tracking  and 
re-routing.  Because  transitory  events  are  unplanned  and  usually  unantici¬ 
pated,  they  can  pose  a  safety  hazard,  particularly  for  the  blind  and  visual¬ 
ly-impaired  traveler.  Transitory  events  are  often  temporary,  disappearing 
in  a  matter  of  minutes  or  hours,  but  they  can  involve  sudden,  unanticipat¬ 
ed  changes  that  have  a  much  longer  presence,  lasting  months  or  even 
years,  depending  on  the  type  of  event  and  the  context  for  the  obstacle  or 

event. 

Figure  2  shows  an  example  of  a  transi¬ 
ent  obstacle  on  a  street  adjoining  our 
local  campus  environment.  The  con¬ 
struction  debris  and  temporary  fencing 
prohibited  everyone  (including  the  dis¬ 
abled)  from  using  the  walkway,  and  for 
a  period  of  approximately  three  days 
caused  local  residents  and  campus  visi¬ 
tors  inconvenience.  Avoiding  the  fenc¬ 
ing  and  debris  in  this  area  entailed 
walking  through  the  roadway,  and  due 
to  the  lack  of  a  nearby  crosswalk  or 
curb  cuts,  presented  a  safety  hazard  for 
all  members  of  the  public. 

Figure  3  shows  a  common  transient  obstacle  in  the  center  of  the  George 
Mason  University  (GMU)  campus  causing  particular  difficulty  to  blind, 
and  visually-  and  mobility-impaired  students.  The  yellow  and  black  elec- 


Figure  2.  Construction 
debris  and  fencing 
temporarily  blocking 
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trical  conduits  covering  the  electrical  cords  are  placed  in  the  walkway  as  a 
means  of  protecting  the  cords,  as  well  as  to  minimize  the  tripping  hazard 
posed  by  the  cords  across  the  walkway.  This  type  of  transient  obstacle  is 
not  of  major  importance  or  concern  to  non-disabled  students,  who  simply 
step  over  it,  but  it  is  a  significant  hazard  for  blind  and  visually-impaired 
individuals,  as  well  as  mobility-impaired  individuals,  whose  wheel  chairs 
cannot  safely  cross  over  the  top. 


Figure  3.  Electrical  cords  and  conduits  across  the  walkways 


The  UCSB  Personal  Guidance  System  was  not  designed  to  -  and  could  not 
-  accommodate  transitory  events  such  as  those  shown  in  Figure  2  and 
Figure  3.  This  limitation  forms  a  major  motivation  for  our  research  pro¬ 
ject. 

The  GMU  campus,  in  striking  similarity  to  the  UCSB  campus,  is  rapidly 
expanding  and  attracts  a  large  number  of  students  with  mobility  and  visu¬ 
al  impairments  (estimated  by  GMU  Assistive  Technology  staff  to  number 
between  250  and  300).  These  students,  and  existing  faculty,  staff,  and 
community  members  also  have  the  misfortune  of  dealing  with  the  con¬ 
struction,  remodeling,  expansion,  and  growth  fostered  by  GMU’s  rapid  ex¬ 
pansion.  The  construction  barricades,  detours,  obstacles,  and  chaos  have 
become  a  very  regular  part  of  campus  life  and  are  simply  ignored  or  side¬ 
stepped  by  able-bodied  students,  faculty,  and  staff.  For  individuals  that  are 
blind  or  mobility  impaired,  this  is  simply  not  possible,  and  the  challenges 
posed  by  these  temporary  inconveniences  are  very  serious  and  deeply  im¬ 
pactful. 
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GMU,  being  faced  with  a  very  similar  set  of  circumstances  and  motiva¬ 
tions,  is  using  contemporary  crowdsourcing  techniques  and  mapping  sys¬ 
tems  to  provide  a  demonstration  and  testbed  environment  for  collecting, 
verifying,  analyzing,  and  displaying  information  about  temporary  obsta¬ 
cles  and  transient  hazards  in  the  campus  environment.  The  ability  to  pro¬ 
vide  real-time  georeferenced  information  about  changes  in  a  local  envi¬ 
ronment  is  seen  as  a  significant  benefit  of  the  geospatial  crowdsourcing 
movement  addressed  in  seminal  papers  by  Goodchild  (2007,  2009)  and 
the  comprehensive  technical  report  from  this  project  last  year.1 

There  really  is  no  question  that  crowdsourcing  and  related  social  media 
technologies  are  transforming  geospatial  data  production  workflows  and 
end-user  participation  in  GIS  activities. 

Several  research  efforts  and  publications  have  addressed  the  ability  of 
crowdsourcing  to  report  major  natural  events,  as  well  as  assist  those  with 
disabilities.  The  United  States  Geologic  Survey’s  efforts  to  use  social  media 
and  crowdsourcing  to  report  earthquakes  is  notable,2 3 4 5^  as  is  Liu  and 
Palen’s  (2013)  summary  and  evaluation  of  map  mashups  and  crowdsourc¬ 
ing  efforts  associated  with  fires,  floods,  and  disease  outbreaks.4  For  the 
disabled  community,  Kremer  (2013)  provides  a  useful  review  of  some 
crowdsourcing  projects  for  accessibility, s  such  as  Ahn  et  al.  (2006)  where 
crowdsourcing  via  gaming  is  used  to  generate  alternate  textual  descrip¬ 
tions  for  web  images  used  by  screen-reading  programs  for  the  blind  and 
visually-impaired.6 7  Takagi  et  al.  (2008)  present  a  similar  social  accessibil¬ 
ity  framework  where  crowdsourcing  is  used  to  enhance  the  accessibility  of 
websites.?  Rice  et  al.  (2012,  2013)  discuss  crowdsourcing  and  accessibility 


1  Rice  et  al.  “Crowdsourced  Geospatial  Data:  A  report  on  the  emerging  phenomena  of  crowdsourced  and 
user-generated  geospatial  data”.  US  Army  Corps  of  Engineers,  November  2012. 
http://www.dtic.mil/dtic/tr/fuiltext/u2/a576607.pdf 

2  “Did  You  Feel  It?"  USGS,  September  7,  2013,  http://earthquake.usgs.gov/earthquakes/dyfi/ 

3  Jason  C.  Young  et  al.,  "Transforming  Earthquake  Detection  and  Science  Through  Citizen  Seismology,” 
The  Woodrow  Wilson  Center,  Commons  Lab,  Science  and  Technology  Innovation  Program,  Case  Study 
Series,  Volume  2  (2013):  1-64. 

4  S.  B  Liu  and  L.  Palen,  “The  New  Cartographers:  Crisis  Map  Mashups  and  the  Emergence  of  Neogeo¬ 
graphic  Practice,”  Cartography  and  Geographic  Information  Science  37,  no.  1  (2010):  69-90. 

5  "Facilitating  Accessibility  Through  Crowdsourcing,”  Karen  m  Kremer,  September  8,  2013, 
http://www.karenkremer.com/kremercrowdsourcingaccessibility.pdf 

6  Luis  von  Ahn  et  al.,  “Improving  Accessibility  of  the  Web  with  a  Computer  Game,”  in  Proceedings  of  the 
SIGCHI  Conference  on  Human  Factors  in  Computing  Systems  (Montreal,  Quebec,  Canada:  ACM, 

2006),  79-82. 

7  Flironobu  Takagi  etal.,  “Social  Accessibility:  Achieving  Accessibility  Through  Collaborative  Metadata 
Authoring,"  in  Proceedings  of  the  10th  International  ACM  SIGACCESS  Conference  on  Computers  and 
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research,  including  how  crowdsourcing  can  be  an  efficient  method  for  in¬ 
corporating  transient  obstacles  and  events  into  mapping  systems.  1>2  These 
publications  and  research  projects  underscore  the  value  of  crowdsourcing 
and  its  applicability  in  a  variety  of  settings,  including  the  reporting  of  tran¬ 
sient  obstacles  as  described  in  this  report. 

In  the  context  of  addressing  the  issue  of  navigation  for  the  disabled,  our 
effort  addresses  three  broader  research  questions:  l)  Is  crowdsourcing  vi¬ 
able  for  monitoring  transitory  events  throughout  their  lifecycle?  2)  What 
quality  control  techniques  are  required  for  monitoring  transitory  events? 
and  3)  Can  authoritative  and  crowdsourced  data  be  integrated  for  transito¬ 
ry  event  reporting?  The  results  of  this  effort  will  not  only  be  applicable  to 
the  university  campus,  but  also  to  broader  concerns  regarding 
crowdsourcing  for  monitoring  any  transitory  events  in  micro¬ 
environments,  such  as  the  military  use  of  crowdsourcing  for  monitoring 
obstacles  in  urban  environments. 

This  funded  research  project  addresses  many  of  these  emerging  phenome¬ 
na  in  a  broad  manner,  and  in  the  current  phase,  develops  a  preliminary 
system  to  gather,  process,  and  display  crowdsourced  information  about 
navigation  obstacles  for  a  campus  environment. 

This  report  will  review  the  development  motivations  and  design  decisions, 
the  recruitment  of  data  contributors,  the  efforts  to  provide  quality  assur¬ 
ance,  and  the  future  directions  of  the  research  effort  in  the  context  of  cur¬ 
rent  achievements  and  challenges.  The  next  section  of  this  report  address¬ 
es  the  broad  development  of  a  geospatial  crowdsourcing  testbed 
environment,  focusing  on  the  requirements  analysis,  overall  system  archi¬ 
tecture,  field  collection  application  development,  quality  control,  proto¬ 
type  development,  recruiting  and  training  activities,  and  usability  consid¬ 
erations. 


Accessibility  (Halifax,  Nova  Scotia,  Canada:  ACM,  2008),  193-200. 

1  Matthew  T.  Rice  etal.,  “Supporting  Accessibility  for  Blind  and  Vision-impaired  People  With  a  Localized 
Gazetteer  and  Open  Source  Geotechnology,’’  Transactions  in  GIS  16,  no.  2  (April  2012):  177-190, 
doi:  10. 1111/j.  1467-967 1.20 12. 013 18.x. 

2  Matthew  T.  Rice  et  at,  “Crowdsourcing  Techniques  for  Augmenting  Traditional  Accessibility  Maps  with 
Transitory  Obstacle  Information,’’  Cartography  and  Geographic  Information  Science  40,  no.  3  (June 
2013):  210-219,  doi:10.1080/15230406.2013. 799737. 
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2  Development  of  a  Testbed  Environment 

In  our  research  proposal  from  2010,  we  outlined  the  general  development 
of  a  crowdsourcing  testbed  environment  that  would  be  used  to  facilitate 
geospatial  data  collection  by  contributors,  study  the  recruitment  of  con¬ 
tributors  and  community  interactions,  and  develop  methods  of  quality  as¬ 
surance  for  crowdsourced  geospatial  data. 

The  development  of  a  GMU  Crowdsourcing  Testbed  Environment  has 
been  a  major  priority  and  center  of  activity  during  this  phase  of  research. 
Beginning  with  only  the  most  basic  conceptual  ideas  about  the  testbed  and 
some  preliminary  demos  (consistent  and  typical  for  proposed  research 
projects),  we  proceeded  to  design  and  develop  a  significant  set  of  tools  for 
the  testbed  environment  and  have  used  these  prototype  tools  for  data  col¬ 
lection  and  analysis.  We  have  made  a  significant  investment  in  data  mod¬ 
eling,  development,  and  contributor  recruitment  and  training  to  facilitate 
the  testbed  environment  and,  more  generally,  the  research  goals  for  this 
project.  This  chapter  discusses  some  of  the  motivations  and  influences  for 
the  testbed,  the  evolution  of  the  testbed  design,  the  development  of  proto¬ 
types  and  web  applications,  recruiting  and  training  activities,  and  modifi¬ 
cations  to  the  testbed  environment  to  enhance  usability. 

Development  and  Evolution  of  General  Technical  Design 

The  development  and  evolution  of  the  general  technical  design  for  our  sys¬ 
tem  began  in  early  2010, 1  when  the  first  conceptual  system  designs  were 
rendered  and  described  by  Rice  et  al.  in  a  publication.  The  current  system 
includes  some  of  these  early  ideas,  but  is  based  on  a  much  more  substan¬ 
tial  data  model  and  design,  led  by  a  requirements  analysis. 

Requirements  Analysis  and  Preliminary  User  Needs  Assessment 

During  the  initial  phase  of  this  research,  we  explored  the  requirements 
and  general  needs  of  the  end-users  of  our  proposed  system.  As  noted  in 
Perkins  (2002),  accessibility  projects  that  adopt  techno-centric  engineer- 


1  MatthewT.  Rice  et  al.,  “Integrating  User-contributed  Geospatial  Data  with  assistive  Geotechnology  Us¬ 
ing  a  localized  Gazetteer,"  in  Advances  in  Cartography  and  GiScience.  Volume  1,  ed.  Anne  Ruas,  Lec¬ 
ture  Notes  in  Geoinformation  and  Cartography  (Springer  Berlin  Heidelberg,  2011),  279-291, 
http://dx.doi.org/10.1007/978-3-642-19143-5_16. 
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ing  solutions,  without  considering  the  social  context  and  needs  of  the  end- 
users,  generally  have  poor  results  and  are  rejected  by  the  community.1 
With  this  in  mind,  we  sought  out  several  future  end-users  of  our  system, 
including  students,  faculty,  and  community  members,  who  are  also  mem¬ 
bers  of  the  disabled  community.  In  discussion  with  them,  a  few  common 
issues  emerged  that  guided  our  data  model  and  system  design: 

1)  The  existing  campus  accessibility  map,  produced  once  a  year  by 
GMU  on  paper  and  in  Portable  Document  Format  (PDF),  is  helpful 
in  designating  Americans  with  Disabilities  Act  (ADA)-compliant 
walkways,  entrances,  and  exits,  but  it  does  not  provide  information 
about  any  temporary  or  current  changes  to  the  campus  accessibility 
infrastructure.  Therefore,  the  map  is  less  relevant  in  supporting 
navigation  activities  due  to  the  construction-related  obstacles  that 
appear  on  a  daily  basis. 

2)  Some  campus  infrastructure  (sidewalks,  automated  ex¬ 
its/entrances,  and  elevators)  have  condition  problems  that  signifi¬ 
cantly  impact  accessibility,  and  the  end-users  would  like  to  have  an 
efficient  way  to  report  and  promote  the  awareness  of  these  items,  so 
that  they  can  be  repaired.  Elevators  that  are  not  working  or  are  tak¬ 
en  out  of  service  without  warning  are  a  significant  problem. 

3)  Sidewalks  with  significant  cracks,  protruding  edges,  or  uplifted  sec¬ 
tions,  as  well  as  pavers  or  bricks  that  are  loose  and  protruding  up¬ 
wards  can  be  a  serious  tripping  hazard  for  blind  and  visually- 
impaired  students.  They  can  also  be  an  obstruction  hazard  for  stu¬ 
dents  with  power  chairs  that  may  not  have  adequate  clearance  due 
to  wheel  size  or  the  presence  of  an  EZ  Lock  device  on  the  bottom  of 
the  power  chair. 

4)  Some  campus  infrastructure  (hydrants,  manholes,  and  curb  cuts) 
have  significant  design  problems  which  limit  accessibility.  End- 
users  would  like  to  have  a  way  to  report  these  issues  and  inform  fu¬ 
ture  design,  and  possibly  to  fix  current  design  problems  through  re¬ 
construction. 


1  Chris  Perkins,  “Cartography:  Progress  in  Tactile  Mapping,"  Progress  in  Human  Geography  26,  no.  4 
(August  1,  2002):  521-530,  doi:10.1191/0309132502ph383pr. 
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5)  Equipment  associated  with  special  events  placed  on  the  walkways 
and  public  areas  present  an  accessibility  challenge.  This  equipment 
includes  tents,  concrete  anchors  and  support  structures  for  tents 
stretching  across  walkways,  and  electrical  conduits  to  support  pub¬ 
lic  address  systems  and  ad-hoc  electrical  cabling  for  special  events. 
These  items  are  a  tripping  hazard  for  the  blind  and  visually- 
impaired,  and  present  many  difficult  clearance  problems  for  the 
mobility-impaired. 

6)  Construction-related  detours  are  frequent,  often  unannounced,  and 
are  marked  in  an  inconsistent  fashion,  with  barrels,  cones,  and  cau¬ 
tion  tape  strung  between  permanent  objects  and  casually-drawn  or 
non-uniform  signage  indicating  the  direction  and  nature  of  the  de¬ 
tour.  There  is  no  consistent  method  for  announcing  and  indicating 
the  frequent  detours  related  to  construction. 

7)  The  surfaces  of  certain  campus  walkways,  crosswalks,  and  public 
areas  are  difficult  with  regard  to  traction,  making  tripping  and  slips 
more  likely,  and  preventing  power  wheelchairs  from  maintaining 
consistent  traction.  This  is  a  byproduct  of  weather  events  (snow, 
ice,  rain,  mud)  and  the  inadequate  design  of  some  walkways  that 
are  consistently  flooded  or  slippery  during  specific  weather  events. 
Snow  clearing  activities,  which  often  block  curb  cuts  are  a  related 
problem.  Pavers  and  bricks  that  are  loose  are  common  in  some  are¬ 
as  of  campus.  These  items  shift  underneath  the  wheels  of  power 
chairs  and  underneath  the  feet  of  pedestrians,  causing  traction 
problems. 

8)  Street  crossing  points  are  not  always  well  marked.  Some  crossing 
areas  require  a  detour  through  the  roadway  to  travel  to  the  nearest 
opposite  curb-cut  or  accessible  entryway. 

9)  Crowds  are  difficult  to  navigate  through,  particularly  when  they  are 
accompanied  by  bicyclists,  skateboarders,  and  vehicles  on  the 
walkways. 

to)  Several  areas  of  campus  and  in  the  local  community  are  common 
parking  areas  for  construction  and  maintenance  vehicles,  and  these 
vehicles  often  block  the  right-of-way. 
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Many  members  of  the  local  disabled  community  (our  system’s  end-users) 
expressed  an  interest  in  reporting  problems.  A  flexible,  simple  interface 
would  be  required  for  facilitating  this  reporting,  particularly  for  end-users 
that  are  visually-impaired  or  blind.  Several  end-users  wanted  to  be  able  to 
communicate  the  nature  of  common  obstacles  to  non-disabled  community 
members,  in  order  to  encourage  understanding  and  to  improve  the  pro¬ 
spects  for  future  accessibility. 

Based  on  this  general  requirements  analysis  and  user  needs  assessment, 
we  developed  a  design  for  our  system  that  would  facilitate  reporting  a  va¬ 
riety  of  transient  events  related  to  obstructions,  detours,  poor  surface  con¬ 
ditions,  crowds  and  events,  entrances,  exits,  and  other  items  needing  re¬ 
pair.  The  system  would  be  map-based,  and  would  be  designed  to  augment 
the  annual  campus  accessibility  map  by  providing  location  and  infor¬ 
mation  about  transient  events  and  obstacles.  The  system  would  facilitate 
the  involvement  of  end-users  in  receiving  information  about  transient 
events  and  obstacles.  Because  many  of  these  end-users  carry  a  mobile 
computing  device  with  them,  the  system  would  need  to  accommodate  a 
variety  of  display  devices  and  the  interface  would  need  to  accommodate 
desktop  and  mobile  screen  sizes. 

A  variety  of  photographic  material  and  documentation  of  the  most  com¬ 
mon  issues  enumerated  above  has  been  gathered  and  continues  to  be 
gathered  by  project  personnel  to  be  used  in  educating  and  informing  non¬ 
disabled  project  contributors.  These  photographs  are  periodically  viewed 
and  prioritized  by  some  of  the  same  end-users  that  participated  in  the  re¬ 
quirements  analysis,  and  subsequently  used  in  our  project  training  activi¬ 
ties. 

Defining  the  Initial  Pool  of  Report  Contributors 

The  recruitment  process  began  by  identifying  who  would  be  the  main  con¬ 
tributors  of  obstacle  reports.  For  our  testbed  on  the  GMU  Fairfax  campus, 
the  potential  contributors  were  expected  to  be  individuals  who  regularly 
navigate  the  campus  environment,  including  students,  faculty,  and  staff. 

For  testing  the  prototype  and  performing  the  initial  evaluation  of  the  con¬ 
tribution  tools,  we  focused  on  recruiting  students  and  faculty  members 
from  our  close  network,  such  as  the  students  and  staff  of  the  Department 
of  Geography  and  Geoinformation  Sciences.  We  invited  professors  and  in¬ 
structors,  graduate  and  undergraduate  students,  Graduate  Research  Assis- 
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tants  (GRAs)  and  Graduate  Teaching  Assistants  (GTAs),  as  well  as  friends 
and  family  that  live  in  the  area  surrounding  the  campus  to  participate  in 
training  activities.  Furthermore,  we  asked  instructors  of  geographic  infor¬ 
mation  systems  (GIS)  courses  (at  both  the  undergraduate  and  graduate 
levels)  to  allow  us  to  provide  a  short  training  session  to  the  students  in 
their  courses.  We  asked  the  instructors  of  those  courses  to  provide  extra 
credit  to  motivate  students  to  complete  the  training,  complete  a  categori¬ 
zation  exercise,  and  submit  reports  using  the  web  interface.  In  total,  there 
were  over  90  participants  recruited  to  undergo  training. 

Architecture 

In  order  to  support  the  needs  of  our  potential  end-users,  as  determined 
during  the  requirements  analysis  and  preliminary  needs  assessment,  a  da¬ 
ta  model  was  developed,  information  collection  interfaces  were  developed, 
and  user  registration  was  planned.  Each  of  these  components  is  described 
below. 

Data  Model  Development 

The  term  “Data  Model”  is  employed  in  many  different  contexts,  with  a 
range  of  meanings.  For  the  work  documented  here,  the  Data  Model  for 
testbed  development  consists  of  a  structured  diagram  with  supporting  text 
that  shows  the  actors,  elements,  and  processes  that  contribute  to  the  ap¬ 
plication  goals,  and  to  the  overall  research  goal  of  assessing  the  value  of 
crowdsourced  geographic  information.  Although  ideally  a  data  model  will 
eventually  be  able  to  serve  several  purposes,  the  context  of  this  work  sug¬ 
gested  that  an  application  goal  of  identifying  obstacles  to  navigation  for 
disabled  persons  in  a  campus  environment  was  appropriate.  A  simplified 
version  of  the  overall  Data  Model  for  testbed  development  as  of  this  writ¬ 
ing  appears  in  Figure  4. 
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Figure  4.  Simplified  data  model  for  testbed  development 


The  process  of  data  model  development  is,  usually,  an  iterative  one,  and 
that  was  the  case  here.  This  process  began  with  the  identification  of  actors 
including  Contributors  and  Reviewers.  The  primary  elements  that  those 
actors  would  generate  and  review,  respectively,  were  reports  of  obstacles 
to  routing  across  campus.  Those  reports  are  combined  to  generate  an  Ob¬ 
stacles  database.  In  subsequent  rounds  of  data  model  iteration  and  re¬ 
finement  the  types  of  reports  to  be  submitted  were  identified,  and  the  ele¬ 
ments  to  be  stored  in  the  obstacles  database  were  identified. 


More  specifically,  the  data  to  be  collected  by  Contributors  and  the  poten¬ 
tial  uses  of  those  data  are  summarized  in  Table  l. 
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Table  1.  Data  Collected  by  Contributors 


Data  Element  Collect¬ 
ed 

Potential  Uses 

ContributorlD /  Username 

•  Tracking  Contributors  for  measuring  productivity  or  reliability 

Report  Type  (New  Obsta¬ 
cle,  Confirmation  of  Obsta¬ 
cle,  Report  Obstacle  Re¬ 
moved) 

•  Used  to  segregate  reports 

•  Potentially  used  for  visualization  of  different  report  types 

•  Used  in  the  Reviewing  /  Quality  Assessment  Process 

Location  (Text/ Ad  dress) 

•  Used  to  display  to  Reviewers  for  confirmation 

•  Potentially  used  to  direct  responders  for  mitigation 

•  Potentially  used  to  direct  End  Users  to  avoid  obstacles 

Latitude/Longitude 

•  Potentially  used  for  confirmation,  mitigation,  and  avoidance 

Report  Date 

•  Used  to  define  when  obstacles  are  First  Reported  and  Last 
Reported 

•  Potentially  used  for  spatio-temporal  analysis 

Obstacle  Type  (Sidewalk 
obstruction,  Construction 
detour,  Entrance  exit  prob¬ 
lem,  Poor  surface  condi¬ 
tion,  Crowd/event,  Other) 

•  Used  for  visualization  of  different  types  of  reports/obstacles 

•  Potentially  used  for  routing  variants  based  on  disability  type 

Image 

•  Potentially  used  for  confirmation,  mitigation,  and  avoidance 

•  Potentially  used  for  manual  checking /  combination  of  re¬ 
ports  into  obstacles 

Duration  (Short,  Medium, 
Long) 

•  Potentially  used  to  direct  response/mitigation  activities 

•  Used  to  compute  temporal  presence 

Urgency  (Low,  Medium, 

High) 

•  Potentially  used  to  direct  response/mitigation  activities 

•  Used  by  Reviewers  to  prioritize  confirmation  activities 

Comments  (location,  ob¬ 
stacle) 

•  Used  for  a  range  of  confirmation  activities 

•  Potentially  used  to  inform  other  Contributors  and  Application 
End  Users 

Status 

•  Automatically  set  to  “1”  or  "Unconfirmed”  until  Review 

Although  the  review/quality  assurance  processes  are  primarily  manual  in 
the  testbed  environment,  each  of  the  data  elements  collected  above  can 
contribute  to  measures  of  completeness  and  consistency.  That  is,  if  a  single 
report  has  all  of  the  potential  fields  completed  by  the  Contributor,  then  it 
may  receive  a  higher  completeness  score,  and  would  generally  be  consid¬ 
ered  to  be  more  reliable.  With  regard  to  consistency,  if  multiple  reports  are 
received  in  close  proximity,  and  there  is  substantial  agreement  between 
the  data  elements  the  combined  reliability  of  these  reports  creates  greater 
confidence  in  the  accuracy  of  the  obstacle.  These  potential  measures  are 
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discussed  in  more  detail  below.  Additionally  there  are  data  elements  such 
as  Report  IDs  that  are  automatically  added  to  reports  for  tracking  purpos¬ 
es. 


Subsequent  to  the  reporting  process,  additional  data  elements  are  collect¬ 
ed  during  the  Review/Confirmation  process  to  be  used  in  generating  the 
Obstacles  database  from  the  collected  reports  (Table  2). 


Table  2.  Review/Confirmation  Elements 


Data  Element  Collect¬ 
ed 

Potential  Uses 

ObstaclelD 

•  Unique  ID  for  Obstacles  used  for  tracking 

ReviewerlD(s) 

•  IDs  of  all  Reviewers  who  have  examined  the  Obstacle 

•  Potentially  used  for  tracking  of  reviewer  performance  and 
computation  of  confidence  measures 

ReportlD(s) 

•  The  IDs  of  all  Reports  that  contributed  to  the  generation  of 
the  Obstacle  used  for  tracking  and  obstacle  history 

ObstacleType 

•  Determined  by  the  Reviewer  from  the  Report(s)  or  by  field 
examination 

•  Uses  as  above 

Estimated  Start  Time 

•  From  initial  report,  duration  estimate,  and  Reviewer  infor¬ 
mation  gathering 

•  Potentially  used  for  spatio-temporal  analysis 

Estimated  End  Time 

•  From  Estimated  Start  time  and  Duration  estimate 

•  Potentially  used  for  spatio-temporal  analysis 

First  Reported 

•  Generated  from  the  collection  of  Reports  that  contribute  to 
the  obstacle 

•  Potentially  used  for  spatio-temporal  analysis 

Last  Reported 

•  Generated  from  the  collection  of  Reports  that  contribute  to 
the  obstacle 

•  Potentially  used  for  spatio-temporal  analysis 

Obstacle  Status 

•  Updated  by  Reviewers  from  initial  setting  of  “Unconfirmed”  in 
reports 

•  Potentially  used  to  track  Obstacles,  identify  malicious  users, 
prioritize  Obstacles  for  further  review 

Obstacle  Urgency 

•  Determined  by  the  Reviewer  from  the  Report(s)  or  by  field 
examination 

•  Uses  as  above 

Obstacle  Image(s) 

•  One  or  more  Images  selected  or  contributed  by  Reviewers 

•  Potentially  used  for  dissemination  to  Contributors  (for  confir¬ 
mation  reports),  Reviewers  (for  creation  of  obstacles),  and 

End  users  for  information  regarding  obstacles  and  their 
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avoidance 

Obstacle  Location 

•  Determined  by  the  Reviewer  from  the  Report(s)  or  by  field 
examination 

•  Uses  as  above 

Obstacle  Lati¬ 
tude/Longitude 

•  Determined  by  the  Reviewer  from  the  Report(s)  or  by  field 
examination 

•  Uses  as  above 

Comments 

•  Collection  of  comments  from  Reviewers,  or  from  Contributors 
as  approved  by  Reviewers 

•  Uses  as  above 

Finally,  data  elements  that  are  generated  by  summarizing  information 
from  the  Reports  Database  and  the  Obstacles  Database  will  be  developed 
in  future  work.  These  may  serve  a  number  of  purposes  potentially  includ¬ 
ing  generating  spatial  databases  of  obstacles,  reporting  on  the  presence  of 
obstacles  over  time  to  decision  makers,  and  reporting  to  authoritative 
sources  regarding  the  mitigation  of  obstacles  across  the  campus  environ¬ 
ment. 

Event  Reporting  Information  Collecting  Requirements 

The  event  reporting  application  developed  in  this  project  was  chosen  to 
meet  the  requirements  of  the  end-users  interviewed  during  the  initial  re¬ 
quirements  analysis  phase  of  the  project,  as  described  above.  Moreover, 
the  information  collected  was  intended  to  instantiate  the  data  model  that 
supports  the  overall  project  goals. 

Toward  those  ends,  contributors  submit  reports  to  add,  edit,  and  confirm 
obstacles.  Reviewers  examine  the  reports  that  are  submitted,  and  use  the 
reports  to  generate  obstacles.  The  obstacles  are  periodically  reviewed  by 
reviewers  and  closed  or  removed  from  view  when  they  are  no  longer  rele¬ 
vant.  The  reporting  application,  while  limited  to  a  small  geographic  area,  is 
similar  in  some  ways  to  the  USGS  National  Map  Corps  Structures  Applica¬ 
tion,  which  is  a  crowdsourcing  program  enabling  users  to  add,  edit,  or  re¬ 
move  data  from  the  USGS’s  National  Map  Database.1 

More  explicitly,  the  information  content  gathered  through  the  event  re¬ 
porting  interfaces  include  location,  obstacle  type,  duration,  urgency,  and 


1  “This  is  the  home  of  The  National  Map  Corps,”  USGS,  September  8,  2013, 
https://mv.usgs.gov/confluence/displav/nationalmapcorps/Home 
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text-based  comments.  The  information  content  associated  with  these 
event-reporting  functions  is  described  below. 

Review  of  Event  Reporting  Interfaces 

In  order  to  make  decisions  about  our  own  application  development  and 
deployment  processes,  several  significant  geospatial  crowdsourcing  appli¬ 
cations  were  reviewed.  Applications  of  this  type  were  previously  profiled 
in  chapter  3  of  the  large  phase  1  technical  report  from  2012  (Rice  et  al. 
2012),  and  separated  into  categories  based  on  their  primary  function 
(Table  3).  The  reporting  applications  profiled  in  the  phase  1  report  include 
Louisiana  Bucket  Brigade,  GasBuddy,  Street  Bump,  Syria  Tracker,  and 
Wikipedia.  Each  application  involves  the  collection  of  data  by  contribu¬ 
tors,  and  in  some  cases,  the  reporting,  display,  and  confirmation  of  contri¬ 
butions. 

To  recognize  the  applications  that  have  been  influential  in  our  work,  we 
profile  Waze1  and  SeeClickFix,2  which  are  popular  tracking  and  reporting 
applications.  Each  of  these  applications  has  a  growing  contributor  base 
and  commendable  interface  design.  In  the  case  of  Waze,  (reviewed  in  the 
phase  1  report  as  a  tracking  application)  the  interface  is  designed  for  use 
by  a  driver  or  passenger  in  a  vehicle,  and  efforts  have  been  made  to  greatly 
simplify  the  mode  of  contribution  and  the  interface. 


Table  3.  Geospatial  crowdsourcing  applications 


Tasks 

Description 

Example 

Imaging 

Building  collections  of  imagery. 

•  Grassroots  Mapping 

Georeferencing 

Rectifying  maps  and  imagery. 

•  Grassroots  Mapping 

•  NYPL  Map  Rectifier 

Transcribing 

Converting  text  resources  to  a  digital  form. 

•  OldWeather 

Digitizing 

Collecting  geospatial  feature  geometry  and 
attributes  from  maps  or  imagery. 

•  OSM 

•  Google  MapMaker 

•  Wikimapia 

Attributing 

Adding  descriptive  information  to  known 
geospatial  features  or  datasets. 

•  Galaxy  Zoo 

Reporting 

Collecting  information  about  a 

•  Louisiana  Bucket 

1  “Get  the  best  route,  every  day,  with  real-time  help  from  other  drivers,"  August  24,  2013, 
"http://www.waze.com 

2  “Report  neighborhood  issues  and  see  them  get  fixed,"  SeeClickFix,  August  22,  2013, 
http://seeclickfix.com 
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location,  usually  through  observation  ora 
mobile  device. 

Brigade 

•  GasBuddy 

•  Street  Bump 

•  SyriaTracker 

•  Wikipedia 

Searching 

Searching  maps  or  imagery  to  identify  spe¬ 
cific  features. 

•  Field  Expedition: 

Mongolia  -  Valley  of  the 
Khans  Project 

•  DARPA  Red  Balloon 

Tracking 

Collecting  paths  and  traces,  usually  using 
GPS. 

•  Waze 

Validating 

Verifying  the  quality  of  existing 
geospatial  information. 

•  NAVTEQ  Map  Reporter 

•  Geo-Wiki.org 

•  OSM  Inspector 

Polling/Surveying 

Collecting  place-based  opinions  or 
information  from  users. 

•  SurveyMapper 

Socializing 

Contributing  geospatially  referenced  in¬ 
formation  to  social  media  sites. 

•  Twitter 

•  Flickr 

•  Foursquare 

Sharing 

Placing  content  on  a  hosted  site, 
potentially  including  data,  applications,  or 
finished  maps,  where  users  can 
access  and  mash-up. 

•  ArcGIS  Online 

•  GeoCommons 
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Figure  5.  Waze  reporting  interface 
and  map  display  (May  2013) 


Waze  (Figure  5)  is  an  Israeli- 
founded  navigation  tool  that 
uses  crowdsourcing  to  estab¬ 
lish  reports  of  traffic  condi¬ 
tions,  police  incidents,  acci¬ 
dents,  hazards,  and  traffic 
cameras.  The  Waze  report¬ 
ing  application  is  significant 
due  to  its  intuitive  interface, 
design,  and  ease  of  use.  Our 
own  efforts  to  reduce  the 
number  of  categories  and 


selection  choices  for  data  contributors  while  maintaining  the  quality  of  in¬ 
formation  is  based  on  our  study  and  use  of  Waze.  The  icon-based  data  con¬ 
tribution  interface  is  a  model  for  the  design  of  the  mobile  contribution 
tools  discussed  in  other  sections  of  this  report,  and  the  simplicity  of  Waze 
is  a  general  motivating  example  for  us.  Finally,  the  differentiation  between 
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the  mobile  map  display  in  the  contribution  tools,  and  the  expanded  map 
view  in  the  desktop  web  interface  echoes  our  own  development  of  our  mo¬ 
bile  contribution  tools  and  desktop  tools  with  different  functionality. 

SeeClickFix  (Figure  6)  de¬ 
scribes  itself  as  a  service 
used  to  “Report  neighbor¬ 
hood  issues  and  see  them  get 
fixed.”  It  relies  on  a  combi¬ 
nation  of  contributor  reports 
and  connections  with  local 
municipal,  commercial,  and 
private  entities  to  provide 
resolution  for  the  reports. 
The  motivations  for 
SeeClickFix  (reporting  to  en¬ 
able  311  services  for  munici¬ 
palities),  and  Waze  (user-engagement  to  facilitate  advertising)  inform  our 
attempt  to  provide  a  mobile  reporting  tool.  In  many  ways,  we  seek  to  com¬ 
bine  the  intuitive  simplicity  of  Waze  and  ease-of-use  while  walking  or 
traveling  with  the  thoroughness  of  the  reporting  system  in  SeeClickFix. 

Prototype  Deployment 

Our  prototype  deployment  has  been  guided  by  exemplar  applications,  our 
requirements  analysis,  a  data  model,  and  the  information  needs  of  our 
end-users.  In  order  to  provide  a  flexible  platform  to  collect  reports  from 
contributors  working  in  the  field  or  from  a  desktop  computer  in  an  office, 
we  decided,  in  Fall  2012,  to  begin  our  prototype  development  and  deploy¬ 
ment  activities  with  a  web  application  to  be  used  through  a  browser.  This 
same  application  could  later  be  modified,  resized,  and  reorganized  to  run 
on  a  mobile  device.  In  many  ways,  this  would  save  time  and  effort  because 
code  from  the  desktop  application  deployment  could  be  re-used  in  the 
mobile  application  deployment.  We  anticipated  that  a  mobile  web  applica¬ 
tion  would  be  used  on  a  variety  of  different  mobile  devices  (iOS,  Android) 
and  would  become  our  primary  collection  tool.  We  planned  on  the  desktop 
web  application  being  both  a  collection  tool  and  a  viewing  and  manage¬ 
ment  tool,  and  in  subsequent  phases  of  the  research  would  adopt  an  ex¬ 
panded  role  as  quality  assurance  methods  became  more  robust  and  new 
ways  of  visualizing  content  were  developed. 


10601-10621  Braddock  Road  Fairfax. 
Virginia 


Red  Light  FAR  Too  Long  (>) 


*  _ 


Actions 

Vote  To  Fix  Issue 
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Add  Comment  /  Photo 

> 

Follow  Issue 

> 

Close  Issue 

> 

Flag  Issue 

> 

(^)  Fairfax,  Virginia 


Figure  6.  SeeClickFix  mobile  reporting 
tool  (May  2013) 


19 


We  serve  our  applications  using  a  Microsoft  Windows  IIS  platform  run¬ 
ning  on  a  Dell  PowerEdge  server  supplied  by  the  GMU  GGS  and  College  of 
Science  IT  departments.  The  server  uses  PostgreSQL  v.9.2  (X64),  PostGIS 
2.0.1  (providing  spatial  extensions  and  capabilities  to  PostgreSQL),  PHP 
5.3.27  (used  to  parse  PHP  files),  FastCGI,  and  Aptana  Studio  3,  which  is  a 
web-programming  tool.  The  code  is  tested  on  a  local  machine  and  then 
moved  to  a  server  where  it  is  expanded,  permissions  set,  and  activated.  We 
are  developing  additional  website  tools  using  the  Drupal  Content  Man¬ 
agement  System,  v.7.22. 

The  core  code  for  the  web  interface  is  available  on  our  project  server,1  with 
a  listing  of  the  base  html  code,  the  extensive  JavaScript  used  for  interac¬ 
tion  and  display  through  the  web  browser,  and  PHP,  which  is  used  to  read 
and  write  records  to  the  database.  This  code  will  be  available  for  public  use 
and  will  be  documented  to  highlight  functionality. 

Web-based  Event  Reporting  Application  Development 

The  event  reporting  application  was  designed  and  developed  from  several 
initial  demonstration  interfaces  and  ideas, 2 3’3  with  functionality  influences 
from  SeeClickFix  and  Waze. 


In  the  current  web  application,  there  are  six 
main  menus,  which  are  numbered  and  facili¬ 
tate  entry  of  reports.  The  menus,  along  with 
their  functionality,  are  presented  here.  Figure 
7  shows  the  map  interface  and  location  icon 
used  for  georegistering  reports.  Figure  8 
shows  the  interface  menu  where  a  contribu¬ 
tor’s  ID  is  entered  and  a  date  and  time  selec¬ 
tion  for  the  report  is  made.  Figure  9  is  our 
obstacle  location  menu,  with  geocoded  location,  latitude  and  longitude 
references,  and  text-based  location  description  functionality.  Figure  10  is 


Figure  7.  Location 
icon 


1  “Testbed  Environment  Code,"  September  8,  2013,  http://geo.gmu.edu/vgi/codebase/ 


2  Aburizaiza,  Ahmad  0.,  and  Matthew  T.  Rice  (2011),  “VGI  and  Geotechnology  for  Supporting  Blind  and 
Vision-Impaired  People  using  a  Localized  Gazetteer”,  F0SS4G:  Free  and  Open  Source  Software  for  Ge¬ 
ospatial,  2011,  Sheraton  Denver  Downtown,  Denver,  Colorado.  Academic  Track  -  Spruce  Room,  Sep¬ 
tember  16,  2011. 

3  Rice  et  al.,  “Integrating  User-contributed  Geospatial  Data  with  assistive  Geotechnology  Using  a  local¬ 
ized  Gazetteer." 
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the  menu  used  to  select  the  obstacle  type,  and  Figure  n  shows  the  menu 
used  for  providing  duration  and  urgency  estimates  for  reports.  A  report 
contributor  can  provide  a  text-based  obstacle  comment,  and  general  feed¬ 
back  in  the  final  menu  (Figure  13)  before  the  contribution  is  reviewed  and 
submitted  (Figure  12).  These  menus  were  developed  using  JavaScript 
within  the  Google  Maps  API,  and  positioned  using  the  placemark  location 
on  the  map. 


Figure  8.  Welcome  menu 
with  contributor  id  and 
date/time  selection 


1  .Welcome  2.0bstacle  Location  3.0bstacle  Type  4.Upload  Image 

1  .Welcome  2.0bstacle  Location  3.0bstacle  Type  4.Upload  Image 

5.Duration  &  Urgency  6.Comments  &  Feedback 

5. 'Duration  &  Urgency  6.Comments  &  Feedback 

Enter  a  Contributor  ID  (6-8  characters): 

Name  of  closest  address  or  building 
research  hall,  george  r 

'  s  limit): 

Submit 

When  did  you  observe  the  obstacle? 

Nottoway  River  Lane,  George  Mason  University, 

Fairfax,  VA  22030,  USA 

LUHudaLontftvk::  3882619,-7730543 

Next-> 

<_Bast  n 

ext  > 

Figure  9.  Obstacle  location 
menu 


1  .Welcome  2.0bstacle  Location  3.0bstacle  Type 
5.Duration  &  Urgency  6.Comments  &  Feedback 


Describe  the  obstacle  (80  characters  limit): 


sidewalk  obstruction 
til.ltUtllll.I.UJjllJI.I.M 
construction  detour 
entrance/exit  problem 
;  poor  surface  condition 
crowd/event 


Figure  10.  Obstacle  type 
menu 


Figure  11.  Obstacle 
duration  and  urgency 
menu 


1  .Welcome  2.0bstacle  Location  3.0bstacle  Type  4.Upload  Image 


5.Duration  &  Urgency  |  6.Comments  &  Feedback 


Obstacle  Comments 


General  Feedback 


Figure  13.  Obstacle 
comments  and  feedback 
menu 


User  ID: 
Report  date: 
Obstacle  type: 


Obstacle  location: 
Describe  the  location: 


guest  user 

Submission  time:8/25/2013 15:27  PM  II  Report  time: 
sidewalk  obstruction 

Nottoway  River  Lane,  George  Mason  University,  Fairfax, 

VA  22030,  USA 

North  side  of  Exploratory  Hall 
Describe  the  obstacle  type:  Construction  barricade  and  caution  tape  across  sidewalk 

Duration: 

Priority: 

Obstacle  Comm 

General  Feedba  Thank  you  for  adding  this  obstacle  report. 

infirm  |  Edit  ] 


Figure  12.  Contributor 
report  summary  with 
confirm  and  edit,  with 
subsequent  submission 
confirmation 
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Mobile  Application  Development 

The  mobile  contribution  tools  (located  at  http://geo.gmu.edU/vgi/m/)  are 
visually  simpler  and  do  not  have  the  full  implementation  of  quality  assur¬ 
ance  and  analysis  tools,  but  will  only  have  what  is  necessary  for  Contribu¬ 
tors  to  submit  reports  and  Reviewers  to  field-check  reports.  The  layout  of 
the  initial  screen  (Figure  15)  is  designed  to  be  similar  to  Waze  (profiled  in 
previous  sections),  and  includes  icons  for  the  six  categories  of  obstacles 
contained  in  the  desktop  application’s  obstacle  type  menu  (Figure  10).  The 
mobile  application  is  designed  from  the  same  codebase  as  the  desktop 
tool,  but  the  sizing  of  the  interface  is  greatly  reduced  and  interface  ele¬ 
ments  are  located  at  the  top  and  bottom  of  the  screens,  as  typical  for  mo¬ 
bile  web  applications  used  on  a  small  screen.  Figure  14  shows  how  the  el¬ 
ements  for  obstacle  location  are  presented  in  simplified  form  and  reduced 
size.  The  mobile  application  development  environment  is  Sencha  Touch 
and  has  been  tested  for  iOS  and  Android  devices. 


q 


Figure  14.  Mobile 
application,  Obstacle 
Location  menu 


Figure  15.  Initial 
contribution  screen  of  the 
mobile  web  application 


User  Registration  Module 

As  noted  above,  the  current  system  (and  the  directions  provided  during 
contributor  training)  instructs  contributors  to  select  an  ID,  which  they  use 
for  reporting.  This  Contributor-ID  is  stored  in  the  database  and  used  to 
check  consistency  between  reports,  thoroughness  of  report  entry,  and  the 
number  of  reports  submitted.  The  next  revision  of  the  web  application,  to 


22 


appear  in  September,  will  include  a  direct  linkage  between  the  Drupal  con¬ 
tent  management  system  and  the  web  application,  which  will  allow  for 
more  detailed  Contributor  tracking  and  reporting,  and  will  facilitate  more 
formal  user  profiles  which  can  be  used  for  community  engagement 
through  the  content  management  system.  Experiments  with  community 
engagement  of  this  sort  were  conducted  with  Blackboard,  an  internal  uni¬ 
versity  content  management  and  curriculum  design  system.  This  was  done 
in  a  closed  setting  with  students  enrolled  in  a  GIS  course  who  were  also 
trained  through  the  same  online  environment  as  an  experiment  in  online 
recruiting,  training,  and  engagement. 

Although  user  registration  mechanisms  in  CGD  projects  are  thought  to  in¬ 
crease  quality  through  accountability,  as  discussed  in  chapter  4  of  Rice  et 
al.  (2012),  user  registration  is  generally  considered  to  be  a  disincentive  in 
many  open  source  communities.  We  would  like  to  maintain  the  ability  to 
make  anonymous  or  guest  contributions  in  the  future,  but  we  recognize 
the  benefit  of  registration  in  terms  of  quality. 

Methods  of  user  motivation  and  engagement  that  have  found  particular 
success  in  a  variety  of  other  projects  include  contributor  badges,  rewards, 
ratings  and  peer  evaluations.  Some  recognition  systems  provide  titles  to 
individuals  based  on  their  participation  or  expertise.  As  an  example,  the 
Old  Weather  site  (reviewed  in  Rice  et  al.  2012)  classifies  individuals  as  Ca¬ 
dets,  Lieutenants,  or  Captains,  based  on  the  number  of  contributions.  Fu¬ 
ture  improvements  to  our  system  will  allow  us  to  better  track  participants 
of  all  kinds,  and  will  allow  us  to  implement  ratings  or  ranking  systems  to 
increase  engagement. 

General  Reporting  Statistics,  Dynamics,  and  Feedback 

Figure  16  shows  that  the  majority  of  contributors  have  only  submitted  a 
single  report,  which  was  an  expected  result  and  not  too  dissimilar  to  the 
dynamics  of  other  crowdsourcing  systems  such  as  Wikipedia,  where  the 
vast  number  of  contributors  make  a  very  small  number  of  reports  and  a 
very  small  proportion  of  Contributors  make  the  largest  number  of  reports.1 
It  is  currently  uncertain  why  most  contributors  only  make  single  reports, 
though  the  extension  of  an  extra  course  credit  for  undergoing  training  and 


1  For  a  graph  of  Wikipedia  edits  by  the  top  10,000  contributors,  see: 
http://upload.wikimedia.Org/wikipedia/commons/d/dc/Top_Wikipedians_edit_distribution%2C_Nove 
mber_2012.svg 
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making  a  report  was  a  factor  in  one  cohort  of  approximately  20  students 
trained  in  the  first  week  of  May  2013.  For  the  other  60-70  trained  Con¬ 
tributors,  the  reason  is  uncertain. 

Wikipedia  has  a  contribution  distribution  that  is  even  more  skewed  with 
very  few  individuals  with  multiple  contributions  and  a  vast  number  that 
have  contributed  only  one  or  two  edits.  We  anticipate  a  less  skewed  distri¬ 
bution,  though  we  now  recognize  that  this  dynamic  of  uneven  user  contri¬ 
butions  is  probably  common  in  many  crowdsourcing  projects. 


1  2  3  4  5  6  More 

Number  of  posts  per  user 


Figure  16.  Contributions  per  user  (all  users) 

Figure  17  shows  the  frequency  of  obstacle  type  within  reports  we  have  re¬ 
ceived.  This  distribution  will  change  in  the  future  with  a  revised  categori¬ 
zation  scheme,  but  at  present  appears  to  be  consistent  with  our  initial  ex¬ 
pectations. 
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Figure  17.  Frequency  of  reports  by  obstacle  type 


We  had  no  preconceived  ideas  about  the  distributions  for  urgency  (Figure 
18)  or  duration  (Figure  19).  We  did  expect,  however,  based  on  our  training 
results,  to  have  few  high  urgency  reports,  which  are  associated  with  obsta¬ 
cles  posing  a  serious  risk  to  safety.  We  did  initially  switch  from  a  larger, 
more  detailed  set  of  duration  and  urgency  categories  to  a  simpler  set  of 
categories  based  on  user  feedback  and  we  may  make  additional  refine¬ 
ments  in  the  future. 
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Urgency 


Figure  18.  Frequency  of  report  urgency  categories 
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Figure  19.  Frequency  of  report  duration  categories 
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Initial  impacts  of  the  event  reporting  prototype 

During  several  outreach  activities  with  Fairfax  City  elected  officials,  trans¬ 
portation  personnel,  planning  and  economic  development  officials,  and 
the  GIS  manager,  the  event  reporting  application  was  demonstrated  and 
several  reports  inside  the  city  were  noted.  One  report  (shown  in  Figure  20) 
notes  a  closed  sidewalk  with  construction  fencing  extending  to  the  curb, 
preventing  access  except  in  the  roadway.  In  this  area,  pedestrians  either 
detour  into  the  actual 
roadway,  cross  to  the 
other  side  of  the  road 
through  traffic  without 
the  benefit  of  a  cross¬ 
walk,  or  backtrack  100 
meters  to  the  nearest 
corner  to  cross.  Subse¬ 
quently  (and  likely  due  to 
simultaneous  reports 
from  residents)  a  sign 
was  placed  both  north 
and  south  of  the  detour 
to  provide  adequate  no¬ 
tice  (Figure  21).  We  are 


"~V  rrra 

* ? £ 

Figure  20.  Sidewalk  closed  and 
detour  into  roadway,  reported  by 
contributor  application,  Obstacle 
Location  menu 
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collecting  anecdotal  infor¬ 
mation  related  to  any  actions, 
such  as  this,  that  may  be  re¬ 
lated  to  authoritative  entities 
using  information  generated 
by  our  reporting  application 
and  will  report  other  activity 
of  this  type  when  it  is  availa¬ 
ble. 


Figure  21.  Signage  added  to 
facilitate  crossing  problem 
associated  with  detour 
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3  Quality  Assessment  Techniques  for 
Crowdsourced  Geospatial  Information 

Background  Regarding  Quality  Measures  and  Assessment  Techniques  in 
CGD 


Although  the  application  goals  for  the  current  testbed  are  to  develop  prac¬ 
tical  tools  for  the  capture  of  obstacles  to  disabled  navigation,  a  fundamen¬ 
tal  research  goal  is  to  determine  the  value  -  or  some  measure  of  the  quali¬ 
ty  -  of  the  information  that  is  collected.  Although  the  terms  “value”  and 
“quality”  are  -  by  their  nature  -  subjective,  this  research  can  benefit  from 
extant  knowledge  in  the  extensive  literature  regarding  quality  assessment. 
The  next  section  reviews  some  pertinent  elements  of  this  large  literature. 
This  is  followed  by  a  description  of  a  Quality  Assessment  Sub-data  model 
intended  to  guide  the  Review  and  Assessment  processes  in  this  research. 
This  is  succeeded  by  an  outline  of  both  currently  implemented  and  future 
research  goals  in  the  assessment  of  reports  and  confirmation  of  obstacles. 
Finally,  discussions  of  report  coverage  and  malicious  content  detection  are 
provided,  with  suggestions  for  future  work. 

Review  of  Quality  Assessment  Literature 

Broadly  speaking,  quality  assessment  is  the  process  of  determining  if  the 
products  of  any  activity  are  sufficient  to  meet  the  needs  of  stakeholders  in 
the  project.  In  the  context  of  crowdsourced  geographic  information,  this 
process  can  include  determining  the  positional,  temporal,  and  attribute 
accuracy  of  the  information,  the  completeness  and  coverage  of  the  data, 
and  more  generally  its  sufficiency  for  any  particular  application. 

For  geospatial  data  produced  by  US  Federal  agencies,  standards  for  quality 
assessment  have  been  developed  and  used,  including  the  National  Map 
Accuracy  Standards  (NMAS)  and  the  National  Standard  for  Spatial  Data 
Accuracy  (NSSDA),  which  is  more  relevant  and  broadly  applicable  to  digi¬ 
tal  data  that  is  not  destined  to  be  used  only  on  a  printed  map.  These  two 
accuracy  assessment  methods  and  their  applicability  to  crowdsourced  geo¬ 
spatial  data  are  described  in  Rice  et  al.  (2012).  One  could  argue  that  these 
standards  are  not  particularly  relevant  to  the  rapidly  changing  geospatial 
data  collection  environment.  In  particular,  these  standards  are  not  well 
suited  for  modeling  the  transient  events  that  are  the  focus  of  this  research. 
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They  do  however  represent  the  milieu  in  which  data  accuracy  is  viewed  by 
a  large  number  of  professional  geographers. 

More  recent  attempts  to  determine  the  accuracy  of  crowdsourced  data  in¬ 
clude  Haklay’s  2010  study  of  OpenStreetMap  (OSM)  data,  which  demon¬ 
strated  that  the  positional  accuracy  of  OSM  roads  data,  when  compared  to 
authoritative  Ordnance  Survey  data,  was  within  6  meters.  Hakley’s  study 
also  concluded  that  there  was  non-uniformity  in  coverage,  with  a  bias  to¬ 
ward  more  affluent  areas.  Concerns  regarding  coverage  are  pertinent  to 
this  research,  given  that  the  presence  of  Contributors  is  not  expected  to  be 
uniform,  nor  is  the  population  of  end-users  uniform  in  space. 

Quality  Assessment  is  about  more  than  just  positional  accuracy,  however. 
It  can  include  lineage,  logical  consistency,  and  completeness,  and  should 
include  issues  such  as  bias  in  coverage  researched  by  Haklay.  Flanagin  and 
Metzger  (2008)  were  early  proponents  of  the  notion  that  lineage  -  or  cred¬ 
ibility  -  had  traditionally  been  associated  with  authoritative  sources,  but 
that  that  assumption  was  set  to  be  continually  challenged  in  the  emerging 
Volunteered  Geographic  Information  (VGI)  environment.  Goodchild  and 
Glennon  demonstrate  the  value  of  crowdsourced  data  supplementing  offi¬ 
cial  data  in  their  2010  discussion  of  crowdsourced  mapping  during  the  re¬ 
cent  California  wildfires.  As  discussed  in  Rice  et  al.  (2012),  the  issues  of 
consistency  and  completeness  are  strongly  related  to  the  larger  issue  of 
“fitness  for  use”.  Does  a  crowdsourced  dataset  have  errors  that  are  signifi¬ 
cant  enough  to  pose  a  risk  and  create  liabilities  for  the  creator?  Is  there  a 
significant  benefit  to  be  gained  by  using  the  crowdsourced  data  in  addition 
to  or  in  the  absence  of  official  data? 

A  number  of  other  significant  new  publications  have  emerged  addressing 
the  issue  of  quality  assessment,  and  were  not  addressed  in  the  previous 
report  (Rice  et  al.  2012).  Of  specific  interest  is  the  work  of  De  Longueville 
et  al.  (2010),  which  extends  fuzzy  analysis  to  this  research  area  by  intro¬ 
ducing  the  notion  of  “Degree  of  Truth”  as  a  measure  of  certainty  (or  con¬ 
versely  vagueness)  regarding  spatial  data.  Goodchild  and  Li  (2012)  suggest 
(among  other  things)  that  crowdsourcing  itself  can  be  used  to  validate  the 
quality  of  data.  That  is,  external  users  can  contribute  both  data  and  quality 
estimates  of  the  data  of  other  contributors  in  the  crowd.  Koukoletsos  et  al. 
(2012)  have  focused  on  completeness,  employing  automated  matching 
methods  for  both  geographic  and  attribute  elements  of  crowdsourced  data. 
Finally,  Ostermann  and  Spinsanti  (2011)  have  endeavored  to  outline  a 
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conceptual  quality  assessment  workflow  for  CGD  that  includes  methods 
for  enriching  the  data  collected,  assessing  quality  and  credibility,  and  ana¬ 
lyzing  the  geographic  nature  of  the  data  that  is  collected. 

While  this  is  not  the  appropriate  forum  to  continue  a  thorough  literature 
review  of  quality  assessment  in  CGD,  it  is  clear  that  issues  of  credibility, 
accuracy,  and  completeness  are  important  common  themes  considered  by 
experts  in  this  research  area.  We  considered  each  of  these  in  our  ongoing 
development  of  review  and  quality  assessment  procedures  for  the  testbed 
environment  presented  here. 

Design  of  Quality  Assessment  Sub-Data  Model 

The  quality  assessment  process  for  the  research  presented  here  is  encapsu¬ 
lated  in  the  Quality  Assessment  Sub-Data  Model  (Figure  22).  This  sub¬ 
data  model  expands  on  the  elements  within  the  overall  Data  Model  de¬ 
scribed  in  Chapter  2  that  are  specific  to  the  reviewing  procedures  neces¬ 
sary  for  building  the  obstacle  database  from  the  raw  reports.  The  primary 
elements  of  the  Quality  Assessment  Sub-Data  Model  are  briefly  explained 
below. 
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Simplified  Review  /  Quality  Assessment  Sub-Data  Model 


Figure  22.  Simplified  quality  assessment  Sub-Data  Model 


Automated  Report  Quality  Assessment 

It  is  envisioned  that  the  report  confirmation  process  will  consist  of  two 
sets  of  procedures,  Automated  Review  and  Manual  Review.  Every  report 
will  be  subject  to  some  Automated  Review  and  Manual  Review  processes. 
Both  processes  contribute  to  the  computation  of  a  Quality  Score  that  is  as¬ 
sociated  with  the  report  and  which  leads  to  Confirmation  or  Rejection  of 
the  report.  This  process  follows  the  structure  of  well-established  methods 
for  address-geocoding,  where  automated  methods  are  used  to  score  candi¬ 
date  locations,  and  manual  review  is  employed  as  necessary  to  supplement 
or  correct  errors  or  omissions. 
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In  the  current  iteration  of  the  Quality  Assessment  Sub-data  model,  there 
are  posited  three  automated  measures  that  could  contribute  to  the  Quality 
Score  for  a  particular  report.  They  are: 

>  Report  Completeness 

>  The  Contributor  Trust  Measure,  and 

>  A  Similarity  Score  if  the  Report  is  flagged  as  a  confirmation  report 
of  an  existing  obstacle 

Report  Completeness  is  simply  a  measure  of  how  much  information  is 
filled  out  on  the  Report.  The  idea  behind  this  measure  is  that,  generally 
speaking,  a  complete  report  with  all  requested  information  may  reflect  the 
reliability,  training,  and  intentions  of  a  contributor.  A  presumption  is  that 
malicious  users  are  less  likely  to  spend  their  time  completing  each  of  the 
requested  data  fields.  Similarly,  if  a  Contributor  accidentally  begins  an  in¬ 
correct  report,  they  will  likely  abandon  it  before  completely  finishing  the 
data  entry  process. Report  Completeness  is  an  imperfect  reflection  of  the 
nature  of  crowdsourced  data,  where  the  entire  process  of  submitting  re¬ 
ports  is  optional.  One  can,  if  they  choose,  enter  a  report  with  very  little  in¬ 
formation  present;  nearly  all  of  the  responses  are  optional  other  than  a  few 
default  selections.  The  logic  behind  this  measure  is  that  the  more  of  these 
optional  items  that  are  included,  the  more  invested  in  the  process  is  the 
Contributor,  and  therefore  the  report  is  more  likely  to  be  of  high  quality. 

The  Contributor  Trust  Measure  is  -  as  the  name  strongly  suggests  -  some 
means  of  distinguishing  the  quality  of  the  Contributors  of  reports.  This 
measure  reflects  the  notion  in  the  literature  for  credibility  or  authority 
among  Contributors.  Currently,  the  trust  measure  is  implemented  as 
simply  a  categorical  value  assigned  to  Contributors  (l,  2,  or  3)  and  is  as¬ 
signed  by  reviewers.  In  future  efforts  this  could  be  changed  to  a  more  re¬ 
fined  score  through  any  number  of  scoring  techniques,  and  that  score 
could,  in  itself,  be  influenced  in  an  automated  way  by  the  number  of  re¬ 
ports  that  are  confirmed  by  Reviewers.  Eventually  a  Contributor  with  a 
high  Trust  Measure  may  be  invited  to  become  a  Reviewer  of  others’  re¬ 
ports. 

Finally,  if  a  Report  is  flagged  by  the  Contributor  to  be  a  confirmation  of  an 
existing  report,  additional  automated  quality  assessment  measures  are  en¬ 
visioned  in  the  sub-data  model.  Although  these  are  still  under  develop¬ 
ment,  they  may  include  measures  of  nearness  (Location  Overlap),  and 
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concurrence  of  obstacle  type,  duration,  and  urgency.  Essentially,  these 
measures  will  determine  to  what  extent  the  confirmation  report  matches 
the  attributes  and  location  of  the  existing  obstacle.  If  the  report  is  con¬ 
firmed,  data  elements  in  the  obstacle  database  will  be  updated  to  reflect 
this  greater  certainty  regarding  the  quality  of  the  obstacle  information  be¬ 
ing  reported. 

Manual  Report  Quality  Assessment 

In  the  Quality  Assessment  Sub-Data  Model  it  is  proposed  that  a  manual 
reviewing  process  will  also  be  implemented.  Depending  on  the  amount  of 
manual  reviewing  resources  available,  some  subset  of  reports  that  have 
been  through  the  Automated  Review  may  also  examined  by  a  trusted  Re¬ 
viewer.  At  the  moment  the  choice  of  reports  to  review  is  left  to  the  Review¬ 
er,  although  reports  that  are  short-lived  (e.g.  moving  vehicles  on  side¬ 
walks,  temporary  crowds)  are  currently  less  likely  to  be  considered  for 
review  since  they  are  likely  to  have  been  resolved  before  a  Reviewer  can 
examine  them.  The  development  of  a  reviewer  report  prioritization  system 
in  the  upcoming  weeks  will  change  the  way  that  reports  are  brought  to  the 
attention  of  reviewers,  with  high  urgency  and  low  duration  reports  being 
given  a  higher  priority  than  others. 

In  the  current  testbed  environment,  Manual  Review  consists  of: 

>  Selecting  a  report  to  review  (Reviewer  Choice) 

>  Optionally  visiting  the  reported  location  in  the  field  to  confirm  or 
collect  additional  data 

>  Editing  reports  to  correct  information 

>  Changing  the  status  of  reports  to  reflect  the  level  of  review  or  con¬ 
fidence 

During  this  process  the  Reviewer  ID  is  appended  to  the  Report  and  the 
Reviewer’s  Trust  Measure  contributes  to  the  overall  Quality  Score.  Cur¬ 
rently  the  trust  measure  is,  again,  a  categorical  measure  of  Reviewer  per¬ 
formance  but  could  be  refined  in  future  work.  Manual  Reviewers  are  re¬ 
sponsible  for  removing  obvious  locational  errors,  obvious  vandalism,  and 
other  clear  errors  in  reporting.  Perhaps  most  importantly  the  Reviewer  can 
change  the  status  of  the  report  to  indicate  if  it  is  under  review,  confirmed, 
or  rejected  as  a  valid  report. 
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Creating  Obstacles 

Creating  Obstacles  in  the  obstacle  database  is  the  purpose  of  the  automat¬ 
ed  and  manual  review  processes.  Since  no  report  -  even  the  most  spurious 
-  is  deleted,  the  outcome  of  the  review  process  is  either  to  confirm  a  report 
or  set  of  reports  as  an  obstacle,  or  to  fail  to  confirm  a  report  as  an  obstacle. 
While  ideally  the  confirmation  process  would  be  based  solely  on  the  Quali¬ 
ty  Score  generated  by  a  combination  of  the  results  of  the  automated  and 
manual  review  processes,  the  reality  at  the  moment  is  that  this  is  largely 
up  to  the  discretion  of  the  Reviewers. 

There  is  currently  functionality  that  allows  Reviewers  to  select  a  subset  of 
reports,  and  combine  them  into  a  confirmed  Obstacle.  In  so  doing,  the 
Reviewer  can  select  which  of  the  reports  is  the  most  reliable,  in  order  to 
initially  populate  the  obstacle  record.  The  Reviewer  may  then  edit  any  el¬ 
ements  that  need  to  be  corrected.  The  ReportIDs  of  the  reports  that  have 
been  combined  into  an  Obstacle  are  recorded.  Similarly,  any  ReporterlDs 
and  ReviewerlDs  associated  with  those  reports  are  also  associated  with  the 
resultant  Obstacle.  A  number  of  fields  (discussed  in  Chapter  2)  of  the  Ob¬ 
stacles  database  are  calculated  based  on  the  input  reports  (e.g.  First  Re¬ 
ported,  Last  Reported). 

Regardless  of  whether  or  not  a  report  is  rejected  or  confirmed,  the  record 
of  the  report  is  preserved  in  the  Report  Database.  By  retaining  rejected  re¬ 
ports,  subsequent  error  checking  can  be  done,  revisiting  of  reports  can  be 
accomplished  if  new  information  is  received,  and  documentation  regard¬ 
ing  rates  of  rejected  reports  can  be  maintained. 

Report  Publication  Process 

The  end  goal  of  developing  the  Obstacle  Database  from  the  Reports  is  to 
deliver  that  database  in  various  forms  for  the  purpose  of  mitigating  obsta¬ 
cles  to  disabled  routing  through  the  campus  environment.  It  is  envisioned 
that  report  publication  could  take  the  form  of  Summary  Documents  to  be 
presented  to  high-level  decision  makers  or  summaries  of  individual  obsta¬ 
cles  (or  subsets  of  obstacles)  to  particular  agents  within  the  university  for 
the  mitigation  of  those  obstacles  (such  as  facilities  management). 

Perhaps  most  importantly,  the  obstacles  can  be  exported  as  geographic 
layers  for  use  in  applications  that  inform  stakeholders  in  various  ways 
about  the  location  and  nature  of  obstacles.  Subsequent  work  will  include 
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the  development  of  routing  applications  to  be  used  by  disabled  persons  for 
the  purpose  of  navigating  across  campus. 

Accuracy  Assessment 

Methods  of  accuracy  assessment  for  the  obstacles  collected  in  this  research 
are  still  under  development,  but  are  being  based  both  on  well-established 
methods  and  novel  measures  of  accuracy. 

Positional  Accuracy  of  Reports 

Measurement  of  positional  accuracy  requires  a  locational  “truth”  from 
which  a  measure  of  variance  away  from  that  truth  can  be  captured.  Essen¬ 
tially,  the  most  trusted  locational  representation  is  the  one  against  which 
all  other  locational  estimates  are  judged.  This  follows  the  methods  em¬ 
ployed  in  Funk  et  al.  (1998)  and  Church  et  al.  (1998)  where  positional  off¬ 
sets  are  represented  by  vectors,  and  where  the  vectors  can  be  converted  to 
error  fields.  The  properties  of  the  vector  offsets  and  error  fields  can  then 
be  analyzed  for  pattern  or  systematic  error,  and  the  spatial  characteristics 
of  the  error  can  be  described. 

In  this  research,  at  the  current  time,  the  crowdsourced  information  is  not 
easily  comparable  to  known  and  trusted  positions  from  which  error  meas¬ 
urements  can  be  made.  In  future  research,  experiments  are  envisioned 
where  a  set  of  obstacles  are  carefully  documented  by  research  personnel. 
Participants  will  be  directed  to  areas  where  they  are  likely  to  encounter 
these  obstacles,  and  when  reports  are  made,  they  will  be  compared  to  the 
known  positions,  and  the  nature  of  the  offsets  will  be  described.  It  is  be¬ 
lieved  that  this  type  of  analysis  can  assist  in  the  refinement  of  data  collec¬ 
tion  procedures,  can  lead  to  improvements  in  training  of  Contributors,  and 
can  help  in  the  development  of  more  precise  reliability  scores  associated 
with  reports. 

Temporal  Accuracy 

The  challenge  of  conducting  rigorous  spatio-temporal  analysis  has  pre¬ 
sented  a  challenge  to  geographers  for  decades.  Given  the  natural  tendency 
of  geographers  to  focus  on  positional  accuracy,  the  measurement  of  tem¬ 
poral  accuracy  has  received  relatively  short-shrift.  However,  recent  studies 
have  examined  the  distributions  of  incidents  in  a  spatio-temporal  solution 
space  (Eckley  and  Curtin,  2012).  Given  that  the  research  presented  here  is 
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focused  on  transitory  events,  and  given  that  transitory  events  have  a  fun¬ 
damentally  important  temporal  nature,  this  is  a  critical  area  for  future  re¬ 
search.  While  we  are  not  prepared  to  evaluate  temporal  accuracy  at  this 
time,  the  comparison  of  estimated  start  time,  end  time,  and  duration  with 
demonstrably  accurate  values  for  those  variables  is  a  critical  area  for  fu¬ 
ture  research. 

Attribute  Accuracy 

Similar  -  if  not  more  challenging  -  difficulties  exist  in  determining  attrib¬ 
ute  accuracy.  Many  of  the  attributes  that  Contributors  are  asked  to  include 
regarding  obstacles  are  highly  subjective  in  nature.  The  determination  of 
the  “truth”  regarding  attributes  such  as  “severity”  or  “duration”  requires 
expert  input;  input  which  varies  even  among  different  experts.  For  this  re¬ 
search  going  forward,  there  will  be  the  development  of  consensus  beliefs 
regarding  obstacle  attributes,  with  the  range  of  expert  opinion  captured 
and  retained.  Although  there  is  some  disagreement  regarding  their  value, 
measures  of  inter-rater  agreement  (such  as  the  kappa  coefficient)  may  be 
employed  in  an  effort  to  determine  the  strength  of  a  consensus  view  re¬ 
garding  an  attribute  value  (Foody  et  al.  2013).  For  cases  such  as  this,  it  is 
arguably  preferable  to  build  a  distribution  of  probable  values.  That  allows 
the  determination  of  an  expected  value  for  the  attribute,  based  on  consen¬ 
sus  expert  opinion.  This  further  allows  a  description  or  computation  of  a 
difference  -  or  variance  -  from  that  expected  value  along  the  distribution 
of  possible  attribute  values. 

Initial  research  into  attribute  accuracy  has  been  conducted  in  order  to  out¬ 
line  issues  in  categorization  during  the  event  reporting  process.  As  part  of 
the  training  sessions  discussed  below,  participants  were  asked  to  complete 
a  categorization  exercise.  The  exercise  was  designed  to  evaluate  how  well 
participants,  both  end-users  and  potential  contributors,  understood  the 
predefined  obstacle  and  urgency  categories.  This  allowed  us  to  gain  addi¬ 
tional  insight  for  improving  both  the  training  material  overall  and  the  cat¬ 
egories  used  within  those  materials  and  the  event  reporting  application 
itself.  The  exercise  consisted  of  categorizing  15  pictures  that  displayed  dif¬ 
ferent  types  of  obstacles,  as  well  as  assessing  the  urgency  and  duration  in 
each  event.  The  pictures  used  for  the  categorization  exercise  were  taken 
from  a  collection  of  pictures  of  obstacles  found  within  and  adjacent  to  the 
GMU  Fairfax  campus. 
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During  our  requirements  analysis  and  preliminary  user  needs  assessment 
(discussed  in  Chapter  2)  end-users  provided  us  with  additional  insight  into 
how  the  perception  of  the  obstacle  types  and  urgency  differ  according  to 
specific  accessibility  needs.  Depending  on  the  disability  of  the  end-user, 
they  provided  different  answers  for  the  urgency  options.  Also,  they  sug¬ 
gested  modifications  to  some  categories  to  fit  obstacles  that  we  initially 
had  not  considered.  Considering  those  differences,  we  are  contemplating 
including  an  additional  option  in  the  reporting  tool,  where  the  contributor 
can  indicate  the  types  of  disabilities  they  believe  would  be  impacted  by  the 
obstacle. 

Coverage 

As  mentioned  in  the  review  of  the  literature  above,  coverage  of  the  opera¬ 
tions  area  is  a  particular  concern  with  crowdsourced  data,  since  there  is  no 
guarantee  that  Contributors  will  report  from  all  parts  of  the  area.  The  con¬ 
cept  of  coverage  can  be  approached  in  a  number  of  ways.  Geometric  analy¬ 
sis  of  existing  reports  or  obstacles  could  identify  the  largest  polygons  that 
are  essentially  “voids”  of  information  in  the  study  area.  Point  pattern  ana¬ 
lytic  techniques  could  describe  the  extent  to  which  the  reports  are  clus¬ 
tered  -  or  conversely  evenly  distributed  -  across  the  region. 

These  purely  spatial  coverage  measures  do  not,  however,  take  into  account 
what  is  perhaps  the  more  important  element  of  coverage... the  coverage  of 
the  end-user  demand  population.  Efforts  to  report  obstacles  that  occur  in 
areas  where  no  disabled  persons  travel  are  not  of  particular  value  for  an 
application  designed  to  improve  routing  for  disabled  persons.  This  is  as¬ 
suming,  of  course,  that  the  reason  for  “voids”  of  disabled  travel  is  not  the 
presence  of  obstacles  themselves.  There  is  a  substantial  body  of  literature 
regarding  ways  in  which  actions  can  be  taken  to  maximally  cover  a  set  of 
demands  in  space,  the  seminal  paper  being  Church  and  ReVelle  (1974). 
More  recent  efforts  that  combine  location  science  techniques  with  GIS  for 
partitioning  areas  for  patrol  have  also  appeared  (Curtin,  Hayslett-McCall, 
and  Qiu  2010).  Future  iterations  of  the  testbed  being  developed  here  may 
well  contain  computations  of  coverage,  perhaps  leading  to  suggested  re¬ 
porting  sub-areas  for  Contributors. 

With  regard  to  the  current  coverage  status  in  the  testbed  environment, 
Figure  23  shows  the  general  distribution  of  reports  for  the  local  area  (with 
one  report  in  Fairfax  County  Vi  mile  east  of  campus  not  shown).  The  re¬ 
ports  are  fairly  broadly  distributed  across  campus,  which  was  a  surprise, 


37 


due  to  our  expectation  that  we  would  see  more  redundant  reporting  in  the 
center  of  campus  near  the  Johnson  Center  and  quads,  where  most  of  the 
students  congregate.  Instead,  we  have  most  of  the  areas  of  campus  and  the 
common  neighborhood  origins  for  campus  visitors  well  represented  in  the 
geographical  distribution  of  reports.  We  note  four  areas  in  black  that  are 
considered  voids  due  to  no  reports  being  made.  These  areas,  with  the  ex¬ 
ception  of  West  Campus,  are  busy  and  filled  with  the  same  activities  that 
generate  other  reports,  and  we  have  no  clear  understanding  of  why  they 
lack  reports.  We  have  not  yet  made  any  effort  to  study  the  time-space  ac¬ 
tivity  patterns  of  our  contributors  because  of  human-subjects  reviewing 
concerns  and  our  concerns  about  contributor  privacy. 
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Figure  23.  Geographic  reporting  voids 


Malicious  Content  Detection 

An  important  consideration  in  any  crowdsourced  content  system  is  the  po¬ 
tential  for  malicious  and  mischievous  content,  which  can  damage  the 
reputation  and  trust  placed  in  the  system.  The  appearance  of  any  mali¬ 
cious  or  mischievous  content  can  exacerbate  existing  concerns  and  fears 
about  quality  and  undercut  efforts  in  a  number  of  ways.  Authoritative  data 
sources  even  have  this  concern,  as  evidenced  by  the  Ordnance  Survey’s 
rapid  reactions  when  the  public  was  faced  with  news  of  fabricated,  inten- 


38 


tionally-false  content  being  inserted  into  data  for  purposes  of  detecting 
copyright  infringement  (Rice  2001,  2005). 

Wikipedia,  perhaps  the  most  prominent  crowdsourced  content  archive  in 
existence,  has  battled  malicious  content  for  years  and  has  a  sophisticated 
approach  mixing  automatic  detection  of  anomalous  content  and  the  man¬ 
ual  intervention  of  editors  and  reviewers,  who  are  trained  to  identify  and 
fix  errors.  OpenStreetMap  has  a  similar  system,  which  combines  scripts  to 
detect  anomalous  content  and  the  intervention  of  editors  and  reviewers. 
These  systems  are  based  on  the  ideas  embodied  in  Linus’s  Law  (reviewed 
in  Chapter  4  of  Rice  et  al.  2012),  which  suggests  that  with  enough  individ¬ 
uals  inspecting  content,  errors  can  be  removed.  The  same  dual  approach 
used  by  Wikipedia  and  OSM  is  being  developed  for  our  testbed  environ¬ 
ment. 

We  are  developing  scripts  to  detect  anomalous  content,  and  training  two 
reviewers  to  inspect  incoming  reports  for  errors  and  malicious  content. 
The  first  element  of  the  automated  scripting  is  a  PHP-based  tool  that 
quickly  searches  incoming  records  and  existing  database  content  for 
matches  to  a  list  of  Google’s  250  banned  terms.1  This  list  of  banned  terms, 
compiled  manually  by  hackers,  is  now  used  and  modified  by  a  variety  of 
firms  to  search  for  objectionable  content  on  their  system,  and  several 
businesses  now  sell  malicious  content  detection  and  bowdlerization2 3  func¬ 
tions  for  use  on  web  pages  where  end-users  have  the  power  to  comment  or 
provide  input.  Our  approach  does  not  currently  involve  bowdlerization, 
but  will  instead  result  in  a  record  being  flagged  and  removed  from  visibil¬ 
ity  in  the  system.  The  training  of  moderators  and  editors  has  also  included 
knowledge  of  this  list  of  words  that  are  objectionable  to  general  audiences, 
and  training  to  detect  other  forms  of  inappropriate  content  related  to  im¬ 
age  content  or  location-based  map  graffiti,  as  noted  in  Rice  et  al.  (2012). 3 

We  have  implemented  some  functions  to  isolate  and  restrict  report  loca¬ 
tions  to  the  GMU  Fairfax  Campus  and  an  approximate  Vi  mile  radius 
around  campus,  which  reflects  the  possible  areas  that  have  been  deter- 


1  “Google  Blacklist  -  Words  That  Google  Instant  Doesn’t  Like,’’  2600,  September  8,  2013, 
http://www.2600.com/googleblacklist/ 

2  Bowdlerization  is  the  replacement  of  offensive  content  with  less  offensive  content,  named  for  Thomas 
Bowdler,  who,  in  the  early  19th  century  published  censored  versions  of  Shakespeare  with  perceived  of¬ 
fensive  content  removed. 

3  See  pages  78-80  of  Rice  et  al.  (2012)  for  examples  of  text-based  map  graffiti  and  other  malicious  or 
mischievous  content. 
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mined  through  the  outreach  activities  to  be  of  interest.  In  some  parts  of 
our  contribution  tools,  the  geographic  restrictions  are  implemented  while 
in  other  areas  they  are  not.  We  will  be  implementing  a  consistent  re¬ 
striction  of  content  to  prevent  out-of-area  contributions,  and  to  identify 
areas  in  our  study  area  where  contributions  are  meaningless,  such  as  the 
Mason  Pond,  and  in  the  public-restricted  forest  areas.  When  contributions 
are  made  in  these  areas,  we  intend  to  have  them  immediately  brought  to 
the  attention  of  the  moderators,  who  will  manually  check  to  determine 
whether  these  contributions  are  a  result  of  an  incorrectly  georegistered  - 
yet  legitimate  -  report,  or  a  mischievous  report  that  can  be  removed. 
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4  Assessment  and  Examination  of  User 
Dynamics,  Motivations,  and  Community 
Interactions 

An  underlying  premise  of  crowdsourcing  is  that  a  group  of  contributors 
exists,  and  this  group  has  the  technical  ability  and  willingness  to  contrib¬ 
ute  content.  Goodchild  (2007,  2009)  argues  that  geographic  crowdsourc¬ 
ing  (or  alternatively,  VGI)  is  unique,  in  that  core  subject  material  being 
contributed  is  not  highly-specialized,  highly-technical,  or  attained  through 
the  process  of  advanced  education  or  professional  licensing.  The  core  sub¬ 
ject  material  of  geographic  crowdsourcing  is  geography,  of  which  every  liv¬ 
ing  person  has  some  expertise,  at  least  on  a  local  or  personal  level. 

Where  our  crowdsourcing  testbed  environment  requires  anything  other 
than  ordinary  observational  skill,  we  provide  training.  The  general  train¬ 
ing  and  recruitment  process  detailed  in  this  report  has  been  ongoing  for 
several  months  both  in  person  and  online.  We  have  some  knowledge  about 
motivations  (some  observed,  some  suggested,  some  anecdotal),  and  we 
have  received  input  from  subject  matter  experts  and  authoritative  partici¬ 
pants.  This  chapter  reviews  the  dynamics  of  users  and  community  mem¬ 
bers  in  our  system,  with  an  emphasis  on  summarizing  current  and  near- 
term  activities. 

Summary  of  Motivations  in  Crowdsourcing 

Creating  a  successful  method  for  end-user  and  contributor  engagement 
may  be  one  of  the  most  important  and  critical  considerations  in  a 
crowdsourcing  application.  The  success  of  these  projects  depends  on  the 
motivations  of  the  crowd  and  their  willingness  to  participate. 

For  many  of  the  emergency  response  scenarios  reviewed  in  Rice  et  al. 
(2012),  and  more  thoroughly  profiled  in  Goodchild  and  Glennon  (2010) 
and  in  Zook  et  al.  (2010),  the  motivations  of  the  crowdsourcing  contribu¬ 
tors  are  generally  ascribed  to  altruism.  For  OSM  (profiled  extensively  in 
Rice  et  al.  2012),  an  initial  motivation  for  participation  was  described  by 
Coast  (2006)  as  a  result  of  resentment  over  the  pricing  and  licensing  prac- 
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tices  of  the  Ordnance  Survey.1  The  motivation  of  contributors  is  often 
more  complex  than  altruism  or  resentment,  and  may  include  a  desire  for 
self-promotion,  a  compulsion  to  fill  gaps  in  areas  that  lack  spatial  cover¬ 
age,  and  a  desire  to  correct  errors,  as  noted  by  Goodchild  (2007).  Jahn 
(2004)  noted  the  similarities  between  users  sharing  navigation  data  and 
Abraham  Maslow’s  hierarchy  of  human  needs.  Coleman  et  al.  (2009)  pre¬ 
sent  an  analysis  of  the  complex  spectrum  of  user  motivations  and  user 
needs,  and  suggest  that  some  of  the  differences  in  user  needs  are  based  on 
the  different  contexts  in  which  they  contribute. 

Outreach  to  Participants 

There  are  a  number  of  participants  in  this  project.  First,  there  are  nearly 
100  individuals  that  have  undergone  training  after  being  identified  or  vol¬ 
unteering  as  potential  contributors.  Second,  there  are  a  large  number  of 
disabled  end-users.  Third,  there  are  many  campus  organizations  that  deal 
with  accessibility.  Finally,  there  are  several  groups  off-campus  in  Fairfax 
City  and  Fairfax  County  that  are  important  and  influential  participants. 
The  discussion  in  this  section  will  focus  on  outreach  activities  to  each  of 
these  groups  and  is  a  summary  of  involvement  from  these  participants. 

Outreach  to  Contributors 

We  have  reached  out  to  a  significant  number  of  potential  non-disabled 
contributors  to  participate  in  training  sessions  and  provide  obstacle  re¬ 
ports  with  our  reporting  tools.  Eighty-eight  students,  staff,  faculty,  and  lo¬ 
cal  residents  participated  in  single  training  sessions  where  they  completed 
an  obstacle  categorization  exercise,  provided  feedback,  and  were  encour¬ 
aged  to  begin  contributing  reports.  Slightly  less  than  half  of  the  training 
participants  contributed  reports.  One  group  of  twenty-six  potential  con¬ 
tributors  received  an  online  version  of  our  training  delivered  through  the 
Blackboard  educational  software  environment,  and  eleven  of  the  twenty- 
six  completed  obstacle  categorization  exercises  and  feedback.  Although 
this  group  had  a  lower  general  participation  rate,  the  online  engagement 
through  an  online  discussion  board  was  useful  in  generating  detailed 
comments,  suggestions,  and  interactions.  These  students  noted  the  im¬ 
portance  of  having  report  contributors  edit,  comment,  and  close  their  re¬ 
ports  (rather  than  leaving  this  function  to  a  moderator);  they  emphasized 
the  importance  of  uploading  obstacle  pictures  (a  feature  that  has  since 


1  Coast,  “OpenStreetMap.” 
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been  implemented);  and  suggested  that  the  report  contributors  be  allowed 
to  view  a  summary  of  each  report  (another  feature  that  has  been  imple¬ 
mented).  Another  group  of  contributors  trained  in  July  2013  encouraged 
us  to  use  report  location  icons  with  colors  to  indicate  status  (a  feature  we 
have  since  implemented).  They  also  suggested  that  we  enable  the  printing 
of  maps  showing  report  locations  with  accompanying  summaries  of  report 
attributes. 

Outreach  to  End-Users 

As  noted  in  Chapter  2,  a  requirements  analysis  and  preliminary  needs  as¬ 
sessment  was  conducted  to  help  inform  us  about  the  potential  system  use 
scenarios.  The  requirements  analysis  involved  detailed  discussions  with 
end-users,  who  are  part  of  the  local  blind,  visually-impaired,  and  mobility- 
impaired  community.  Three  student  end-users,  one  faculty  end-user,  one 
community  member,  and  several  GMU  staff  who  work  with  end-users 
were  interviewed  for  the  requirements  analysis  and  user  needs  assess¬ 
ment.  During  this  assessment,  each  end-user  cited  the  frustration  of  en¬ 
countering  obstacles  while  navigating  across  campus  or  nearby  neighbor¬ 
hood  areas,  and  while  driving  their  power  chairs.  The  list  of  important 
issues  noted  in  Chapter  2  is  being  updated  when  new  information  is  pre¬ 
sented,  as  a  byproduct  of  continued  outreach  and  contact  with  these  stu¬ 
dents,  faculty,  staff,  and  community  members. 

Outreach  to  Authoritative  Groups 

The  relationships  between  authoritative  participants  in  a  project  like  this 
can  be  difficult  due  to  conflicting  goals  and  interests.  We  are  cognizant  of 
the  difficulties  posed  by  drawing  together  the  groups  of  authoritative  par¬ 
ticipants  described  here  and  we  will  continue  to  conduct  our  outreach  ac¬ 
tivities  with  them  in  a  careful  manner. 

Insights  from  Subject  Matter  Experts 

Several  subject  matter  experts  were  interviewed  and  have  offered  contri¬ 
butions  to  our  project.  Some  of  them  were  contracted  as  consultants,  some 
are  members  of  the  disability  community,  and  some  are  authoritative  fig¬ 
ures  working  for  campus  organizations  that  deal  with  accessibility.  A 
common  thread  through  many  of  their  interviews  is  general  frustration 
dealing  with  inaccessible  environments,  and  advice  to  engage  with  system 
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end-users  (members  of  the  disabled  community)  who  will  be  able  to  intel¬ 
ligently  advise  our  development  decisions. 

Dr.  James  Marston,  a  long-term  resident  of  Santa  Barbara,  California  and 
a  former  employee  of  the  Veteran’s  Administration  Hospital  in  Atlanta, 
suggests  that  frustration  with  inaccessibility  is  very  common.  Citing  his 
experiences  on  the  UC  Santa  Barbara  Campus  and  in  Atlanta,  he  suggested 
that  if  participation  in  the  project  led  to  noticeable  accessibility  improve¬ 
ments,  this  would  be  a  significant  motivation  for  end-users  to  become  en¬ 
gaged  with  the  project.  As  a  visually-impaired  person,  Dr.  Marston  became 
frustrated  with  construction  and  maintenance  vehicles  being  driven  and 
parked  on  walkways  on  the  UC  Santa  Barbara  campus.  This  frustration 
continued  to  fester  until  he  was  given  a  position  on  a  campus  accessibility 
board,  which  allowed  him  the  latitude  and  authority  to  request  changes.  In 
a  recent  report  for  the  project,  Dr.  Marston  suggests  that  we  prioritize  en¬ 
gaging  end-users  in  contributing  reports.  As  end-users  become  contribu¬ 
tors  to  the  project,  they  will  gain  a  similar  feeling  of  empowerment  that 
could  be  useful  in  building  support  and  acceptance  by  authoritative  groups 
that  work  with  disabled  students. 

Dr.  Michael  Goodchild,  echoing  a  sentiment  expressed  in  Perkins  (2002), 
suggested  during  early  phases  of  development  that  the  best  advice  on  sys¬ 
tem  development  would  come  from  eventual  system  end-users,  and  that 
these  individuals  should  be  interviewed  and  questioned  about  their  poten¬ 
tial  uses.  He  suggested  that  a  technology-centered  approach,  without  the 
input  of  end-users,  would  not  work  well.  Goodchild’s  2009  paper  on  the 
phenomena  of  VGI  notes  that  the  most  successful  crowdsourcing  systems 
are  likely  to  be  hybrid  systems  that  mix  the  input  of  subject  matter  experts 
and  authoritative  entities  with  the  expertise  of  the  end-users  and  contribu¬ 
tors  to  the  system.  His  suggestion  to  begin  by  engaging  with  end-users 
was  advice  that  we  took,  undertaking  the  requirements  analysis  and  pre¬ 
liminary  user  needs  assessment  shortly  afterward. 

On-Campus  Groups 

The  GMU  Vice-President  for  Facilities  has  pledged  public  support  for  the 
project  and  has  given  project  personnel  permission  to  ask  for  any  re¬ 
sources  that  are  needed.  This  support  was  given  during  a  public  discussion 
with  Fairfax  City  officials  who  were  already  familiar  with  the  project.  The 
GMU  Vice-President  for  Facilities  reviewed  the  report  contribution  tools 
and  previewed  several  reports  about  blocked  sidewalks  along  Patriot  Circle 
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near  the  Facilities  Buildings,  and  provided  some  real-time  validation  of  the 
reports.  We  envision  the  GMU  Physical  Facilities  group  to  be  a  future  con¬ 
tributor  of  reports  to  our  system  and  have  extended  this  opportunity  to 
them. 

The  Assistive  Technology  Initiative  Office  (ATI)  is  partially  responsible  for 
producing  campus  accessibility  maps  for  the  GMU  Office  of  Equity  and 
Diversity  Services,  which  handles  American  with  Disabilities  Act  (ADA) 
compliance  issues  on  campus.  The  ATI  Office,  with  a  somewhat  overlap¬ 
ping  mandate  with  the  Facilities  GIS  and  mapping  group,  has  faced  diffi¬ 
culty  sharing  data  and  working  toward  common  GIS  and  mapping  goals. 
While  both  groups  have  been  comfortable  for  us  to  work  with  because  of 
our  slightly  different  focus  (academic  research  and  education  rather  than 
practical  systems  implementation),  these  groups  have  differing  goals  and 
perspectives.  We  will  continue  with  outreach  activities  to  both  groups. 

We  have  offered  to  begin  quarterly  meetings  as  part  of  a  new  Accessibility 
Working  Group,  involving  project  personnel,  ATI  staff,  ADA  compliance 
staff,  Facilities  staff,  Office  of  Disability  Services  staff  (who  have  been  the 
subject  of  successful  outreach  for  a  long  time),  and  the  GMU  Kellar  Insti¬ 
tute  for  Human  Disabilities. 

Off-Campus  Groups 

The  GMU  campus  is  bordered  by  Fairfax  City  and  Fairfax  County  (Figure 
24).  Fairfax  City  officials,  including  the  Mayor,  several  City  Council  Mem¬ 
bers,  the  City  Transportation  Director,  the  City  Public  Works  Director,  and 
the  GIS  Manager,  have  all  been  briefed  on  the  project.  Fairfax  City  officials 
view  the  project  as  a  positive  way  to  highlight  efforts  towards  ADA- 
compliance  and  to  strengthen  the  GMU-City  Town/Gown  relationship  that 
has  been  poor  in  the  past.  Base  data  for  the  parcels,  roads,  sidewalks,  and 
other  infrastructure  has  been  contributed  to  the  project  as  a  show  of  sup¬ 
port.  We  also  envision  the  City  of  Fairfax  Public  Works  department  and 
the  GIS  Manager  to  be  report  contributors,  and  have  extended  this  oppor¬ 
tunity  to  them.  We  also  envision  them  using  our  system  to  view  reports 
and  make  changes  to  the  accessibility  issues  in  the  City  of  Fairfax. 
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Figure  24.  Local  political  entity  boundaries  superimposed  on 

reporting  interface 

Several  Fairfax  County  officials  have  also  been  contacted  and  are  support¬ 
ive,  though  more  difficult  to  interact  with  because  of  the  size  of  the  County 
and  the  difficulty  of  addressing  local  issues  in  such  a  large  political  entity. 
Assistants  to  Supervisor  for  the  Braddock  District  have  been  contacted 
through  the  GMU-City  Town/Gown  group. 

The  Virginia  State  Delegate  for  the  local  area  has  been  briefed  on  the  pro¬ 
ject  and  is  supportive.  He  is  familiar  with  GIS  and  hires  GIS  interns  peri¬ 
odically  in  his  capacity  as  an  environmental  consultant,  so  his  support  is 
based  not  only  on  political  factors  but  also  on  enthusiasm  for  the  geospa¬ 
tial  domain. 

The  Fairfax  County  Public  Works  and  Environmental  Services  Director  has 
also  been  briefed  about  the  project  and  is  fully  supportive.  He  has  been  pe¬ 
ripherally  involved  in  an  ADA-related  consent  decree  process  between  the 
US  Department  of  Justice  and  Fairfax  County,  and  offered  an  introduction 
to  County  staff  that  has  been  involved  in  the  remediation  activities  to  satis¬ 
fy  the  consent  decree.  He  is  also  useful  as  a  facilitator  with  other  groups  in 
the  County  that  have  a  mission  related  to  mapping  and  accessibility,  and 
has  offered  introductions  to  several  other  groups  that  use  GIS  in  the  Coun¬ 
ty  government. 
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Some  members  of  the  Fairfax  County  Department  of  Planning  and  Zoning 
have  expressed  interest  in  the  project  and  have  offered  help,  due  to  their 
interest  in  mapping  and  GIS  as  well  as  their  interest  in  code  compliance 
and  ADA  issues.  Outreach  activities  to  this  group  are  ongoing  and  being 
facilitated  by  a  member  of  the  Planning  and  Zoning  Department  acquaint¬ 
ed  with  GMU  project  personnel. 

Recruiting  and  Training  Report  Contributors 

The  recruiting  and  training  process  is  an  essential  step  for  testing  and  as¬ 
sessing  the  quality  of  the  reporting  tools,  and  is  significant  way  to  increase 
the  quality  of  the  information  we  collect. 

There  is  very  little  peer-reviewed  literature  about  training  and  recruiting 
best  practices  within  geo-crowdsourcing.  As  a  supplement,  we  looked  at 
some  of  the  training  methods  and  resources  provided  by  two  successful 
and  popular  applications  that  use  crowdsourcing  to  collect  data:  Waze1 
and  OpenStreetMap  (OSM).2 3 4 5 

Waze  provides  users  with  short  videos  to  explain  the  purpose  and  func¬ 
tionality  of  their  application.  3.  In  addition  to  the  videos,  Waze  provides  a 
user  manual.  The  Waze  training  material  is  short,  simple,  and  comprehen¬ 
sive  enough  to  explain  the  essential  functionality  of  the  system. 

The  OpenStreetMap  training  material  is  not  as  succint  and  directly 
engaging  as  the  Waze  training  material,  and  this  is  certainly  a  reflection  of 
the  greater  technical  complexity  and  scope  of  the  OSM  project.  OSM 
provides  introductory  emails  with  information  about  participation, 
videos, 4  a  wiki,s  and  a  questions  and  answers  site.6  Since  there  are  many 
different  ways  to  contribute  data  to  OSM,  the  training  methods  vary  in 
length  and  complexity.  Some  OpenStreetMap  training  material  is 
technical  in  nature  with  a  more  rigorous  scientific  theme  than  the  Waze 


1  “Waze  -  Social  Traffic  &  Navigation  App,”  http://www.waze.com/,  accessed  September  9,  2013 

2  “OpenStreetMap,”,  http://www.openstreetmap.org/,  accessed  September  9,  2013 

3  Waze  Gps,  “Waze  Map  Editor  Guide  Full  Clip  |  Waze,"  YouTube,  April  30,  2012, 
http://www.youtube.com/watch?v=HVksbblZ4SQ. 

4  [581]  Steve,  “OpenStreetMap,”  ShowMeDo,  2008, 
http://showmedo.com/videotutorials/series?name=mS2PlZqS6. 

5  “Beginners’  Guide  -  OpenStreetMap  Wiki,”  July  28,  2013, 
http://wiki.openstreetmap.org/wiki/Beginners%27_Guide. 

6  “OpenStreetMap  Help  Forum,”  accessed  August  26,  2013,  https://help.openstreetmap.org/. 
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training  material.  We  want  to  create  material  that  combines  the 
simplicity  and  engaging  nature  of  the  Waze  training  material  with  the 
detailed  explanations  and  thoroughness  of  the  OSM  training  material. 

Our  current  set  of  training  material  includes  an  introduction  to  the  pro¬ 
ject,  examples  of  crowdsourcing  applications,  a  summary  of  the  basic  re¬ 
porting  system  functionality,  and  examples  of  obstacle  types,  obstacle  ur¬ 
gency,  and  obstacle  duration.  After  conducting  training  with  the  first  30- 
35  participants  and  receiving  feedback,  we  implemented  changes  in  our 
training  procedures  to  include  obstacle  duration  and  urgency  information 
During  this  early  training  period,  we  also  modified  the  photographs  used 
for  providing  obstacle  examples,  due  to  confusion  expressed  by  training 
subjects. 
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5  Summary  and  Future  Directions 

For  blind,  visually-impaired,  and  mobility-impaired  persons,  there  are  two 
major  obstacles  to  full  participation  in  society:  information  barriers,  and 
movement  barriers.  These  two  barriers,  described  by  Dr.  Reginald 
Golledge  during  a  2001  commencement  address  at  Simon  Fraser  Universi¬ 
ty,  have  become  more  significant  due  to  the  rapid  evolution  of  digital  in¬ 
formation  channels  and  the  extent  to  which  these  channels  are  inaccessi¬ 
ble.  Systems  developed  to  overcome  these  barriers,  such  as  the  UCSB 
Personal  Guidance  System,  are  a  significant  advancement,  but  even  these 
systems  lack  a  crucial  feature:  the  ability  to  accommodate  the  unplanned 
and  transient  barriers  that  significantly  impact  navigation.  A  sensible  way 
to  capture  and  map  information  of  this  type  is  through  crowdsourcing:  a 
rapidly  emerging  and  evolving  part  of  the  contemporary  information  land¬ 
scape. 

This  research  project  seeks  to  develop  methods  to  crowdsource  the  loca¬ 
tion  and  attributes  of  navigation  obstacles,  and  to  develop  methods  for  val¬ 
idation  and  quality  assurance  that  will  provide  some  confidence  in  our  sys¬ 
tem  and  the  information  it  produces.  The  two  phases  of  this  research  have 
been  productive  and  useful  in  understanding  how  crowdsourced  geospatial 
data  is  evolving  and  being  used  in  society,  and  in  the  development  of  a 
testbed  environment  for  continued  study  of  the  dynamics  and  methods  for 
crowdsourcing.  We  have  implemented  a  training  and  recruitment  pro¬ 
gram,  and  have  created  a  map-based  reporting  tool  in  desktop  and  mobile 
versions  that  has  been  tested  and  used  by  a  number  of  contributors  to  re¬ 
port  obstacles  on  the  GMU  Campus.  We  have  developed  a  comprehensive 
data  model  to  guide  all  of  our  workflows  and  activities,  and  we  are  imple¬ 
menting  practical  methods  to  field  check,  validate,  and  quality  assess  the 
reports  that  are  being  made,  and  turn  those  reports  into  Obstacles. 

There  are  a  number  of  significant  lessons  we  learned  in  the  progress  of  this 
research  project,  and  many  more  that  we  anticipate  learning  in  the  next 
phase  of  our  work.  First,  building  a  community  of  contributors  is  difficult. 
With  approximately  12  active  contributors  from  the  88  we  have  trained, 
we  need  to  train  at  least  220  more  students,  faculty,  staff,  and  other  poten¬ 
tial  contributors  to  achieve  our  contributor  goals,  which  are  40-50  con¬ 
sistent,  active  contributors.  A  factor  that  could  contribute  to  a  higher  re- 
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tention  rate  of  contributors  is  a  more  robust  system  for  engagement,  which 
we  will  be  implementing  in  Drupal  during  September,  and  a  way  of  doing 
unobtrusive  follow-up  communication  with  contributors  that  have  been 
trained.  Creating  incentive-based  games,  rewards,  and  other  similar  sys¬ 
tems  will  be  discussed  during  the  upcoming  months,  with  input  from  other 
faculty  colleagues  that  have  researched  the  efficacy  of  ratings  systems  for 
user  engagement. 

Crowdsourcing  projects  such  as  this  involve  data  collection  by  non¬ 
experts,  which  can  result  in  problems  such  as  non-standard  descriptions, 
errant  placement  of  information,  and  the  differences  in  interpreting  cate¬ 
gories,  locations,  durations,  and  estimates  for  similar  events.  These  prob¬ 
lems  are  a  part  of  the  domain  and  should  be  anticipated.  To  the  extent 
possible,  we  will  work  to  simplify  and  reduce  the  complexity  of  our  report¬ 
ing  tools  to  reduce  errors  in  the  contribution  process.  We  will  continue  de¬ 
veloping  methods  to  assess  the  quality  of  the  information  that  we  collect 
through  the  help  of  our  contributors,  end-users,  and  authoritative  part¬ 
ners. 

In  the  upcoming  months,  we  will  expand  our  training  and  recruiting  pro¬ 
gram  and  capture  several  hundred  more  contributors.  We  will  study  the 
spatial  patterns  of  reports,  identify  and  refine  our  understanding  of  geo¬ 
graphical  data  voids  and  attempt  to  fill  them.  We  will  finish  the  implemen¬ 
tation  of  our  content  management  tools  and  with  their  help;  we  will  in¬ 
crease  the  interaction  among  users.  We  hope  to  see  increased  engagement 
and  retention  of  contributors.  Finally,  we  will  continue  our  community¬ 
building  and  outreach  activities,  and  our  cooperative  work  with  campus 
and  local  authoritative  entities. 

The  next  phase  of  our  research  project  will  include  methods  for  visualizing 
hybrid  collections  of  authoritative  and  asserted  content,  and  associated 
quality  measurements. 
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